Compression and decompression system with reversible wavelets and lossy reconstruction

ABSTRACT

A method and apparatus for encoding and decoding data that includes generating transformed signals in response to input data. In one embodiment, the transformed signals are generated using a reversible wavelet transform. The present invention also includes a method and apparatus for compressing the transform signals into data representing a losslessly compressed version of the input data. In one embodiment, the present invention decomposes the input data using reversible wavelet transforms.

This application is a continuation-in-part of application Ser. No.08/643,268, entitled “Compression/Decompression Using ReversibleEmbedded Wavelets”, filed May 3, 1996, which is a continuation-in-partof application Ser. No. 08/498,036, entitled Reversible WaveletTransform and Embedded Codestream Manipulation, filed Jun. 30, 1995,which is a continuation-in-part of application Ser. No. 08/310,146,entitled Apparatus for Compression Using Reversible Embedded Wavelets,filed Sep. 20, 1994.

FIELD OF THE INVENTION

The present invention relates to the field of data compression anddecompression systems; particularly, the present invention relates to amethod and apparatus for lossless and lossy encoding and decoding ofdata in compression/decompression systems.

BACKGROUND OF THE INVENTION

Data compression is an extremely useful tool for storing andtransmitting large amounts of data. For example, the time required totransmit an image, such as a facsimile transmission of a document, isreduced drastically when compression is used to decrease the number ofbits required to recreate the image.

Many different data compression techniques exist in the prior art.Compression techniques can be divided into two broad categories, lossycoding and lossless coding. Lossy coding involves coding that results inthe loss of information, such that there is no guarantee of perfectreconstruction of the original data. The goal of lossy compression isthat changes to the original data are done in such a way that they arenot objectionable or detectable. In lossless compression, all theinformation is retained and the data is compressed in a manner whichallows for perfect reconstruction.

In lossless compression, input symbols or intensity data are convertedto output codewords. The input may include image, audio, one-dimensional(e.g., data changing spatially or temporally), two-dimensional (e.g.,data changing in two spatial directions (or one spatial and one temporaldimension)), or multi-dimensional/multi-spectral data. If thecompression is successful, the codewords are represented in fewer bitsthan the number of bits required for the uncoded input symbols (orintensity data). Lossless coding methods include dictionary methods ofcoding (e.g., Lempel-Ziv), run length encoding, enumerative coding andentropy coding. In lossless image compression, compression is based onpredictions or contexts, plus coding. The JBIG standard for facsimilecompression (ISO/IEC 11544) and DPCM (differential pulse codemodulation—an option in the JPEG standard (ISO/IEC 10918)) forcontinuous-tone images are examples of lossless compression for images.In lossy compression, input symbols or intensity data are quantizedprior to conversion to output codewords. Quantization is intended topreserve relevant characteristics of the data while eliminatingunimportant characteristics. Prior to quantization, lossy compressionsystem often use a transform to provide energy compaction. JPEG is anexample of a lossy coding method for image data.

Recent developments in image signal processing continue to focusattention on a need for efficient and accurate forms of data compressioncoding. Various forms of transform or pyramidal signal processing havebeen proposed, including multi-resolution pyramidal processing andwavelet pyramidal processing. These forms are also referred to assubband processing and hierarchical processing. Wavelet pyramidalprocessing of image data is a specific type of multi-resolutionpyramidal processing that may use quadrature mirror filters (QMFs) toproduce subband decomposition of an original image. Note that othertypes of non-QMF wavelets exist. For more information on waveletprocessing, see Antonini, M., et al., “Image Coding Using WaveletTransform”, IEEE Transactions on Image Processing Vol. 1, No. 2, April1992; Shapiro, J., “An Embedded Hierarchical Image Coder Using Zerotreesof Wavelet Coefficients”, Proc. IEEE Data Compression Conference, pgs.214-223, 1993. For information on reversible transforms, see Said, A.and Pearlman, W. “Reversible Image Compression via MultiresolutionRepresentation and Predictive Coding”, Dept. of Electrical, Computer andSystems Engineering, Renssealaer Polytechnic Institute, Troy, N.Y. 1993.

Quantization of wavelet coefficients results in pleasing images becausewavelet filters are overlapped. In contrast, quantization ofblock-based, non-overlapped transform coefficients suffer from artifactsat boundaries that are not pleasing.

Removing blocking artifacts from traditional block-based transforms suchas the DCT is difficult for many reasons. The boundary affects everycoefficient. Each transform coefficient affects many spatial domainlocations. Smoothness in the transform domain involves complexrelationships between many coefficients. For example, the class of all2D linear ramps in the spatial domain does not have a nice DCTrepresentation. It is difficult to combine transform domain smoothnessconstraints and quantization constraints. Typically, iterative solutionsare used, with an enhancement operation (smoothing/de-ringing/edgeenhancement) being performed in the spatial domain followed by limitingin the transform domain to keep the enhancement within the quantizationerror.

Edges are often the cause of artifacts in an image. Edge extraction iswell known in the art as a means to identify edges and to remove theedges from an image. The use of gaussians in edge detection has a longhistory. For instance, see E. Marr and E. Hildreth, “Theory of EdgedSection,” Proc. R. SOC. London, Vol. 207, pp. 187-217, 1980; V. T. andThomaso A. Poggio, “On Edge Detection,” IEEE Trans. on Pattern Analysisand Machine Intelligence, Vol. 8, pp. 147-63, 1986; L. Basano, D.Caprile, et al., “Edge-Detection Schemes Highly Suitable for HardwareImplementation,” J. Opt. Soc. Am., Vol. 5, pp. 1170-1175, 1988; and S.Mallate and S. Zhong, “Characterization of Signals from MultiscaleEdges,” IEEE Trans. on Pattern Analysis and Machine Intelligence, Vol.14, pp. 710-732, 1992. However, even though these methods for detectingedges existed in the prior art, there is always a desire to improve theedge detection so that reconstructions can be made with sharp edgeswithout artifacts near the edges.

Compression is often very time consuming and memory intensive. It isdesirable to perform compression faster and/or with reduced memory whenpossible. Some applications have never used compression because eitherthe quality could not be assured, the compression rate was not highenough, or the data rate was not controllable. However, the use ofcompression is desirable to reduce the amount of information to betransferred and/or stored.

SUMMARY OF THE INVENTION

A method and apparatus for performing reconstruction is described. Themethod in the present invention provides for receiving DS and DDcoefficients affected by a boundary, reconstructing SD coefficients tobe smooth across the boundary, applying a vertical inverse transformindividually on each tile, reconstructing D coefficients to be smoothacross the boundary, and applying a horizontal inverse transformindividually on each tile.

BRIEF DESCRIPTION OF THE DRAWINGS

The present invention will be understood more fully from the detaileddescription given below and from the accompanying drawings of variousembodiments of the invention, which, however, should not be taken tolimit the invention to the specific embodiments, but are for explanationand understanding only.

FIG. 1 shows the context dependent relationships in which Children areconditioned on their parents.

FIG. 2 is a block diagram of one embodiment of a compression system ofthe present invention.

FIGS. 3 and 4 illustrate possible geometric relationships of the contextmodel for each bit of each bit-plane in the binary style.

FIG. 5 illustrates a tiled image.

FIG. 6 illustrates importance level entry points, main header syntax.

FIG. 7 illustrates importance level entry point, tile head syntax.

FIG. 8 illustrates importance level locators, main header syntax.

FIG. 9 illustrates importance level locators, tile header syntax.

FIG. 10 illustrates the bit depths of the various coefficients in atwo-level TS-transform and TT-transform decomposition from an inputimage with b bits per pixel.

FIG. 11 is one embodiment of the multipliers or alignments for thefrequency band used for coefficient alignment in the present invention.

FIG. 12 illustrates examples of bit significance representation.

FIG. 13 shows the neighborhood coefficients for every coefficient of acoding unit.

FIG. 14 illustrates a Child based scan order.

FIG. 15 illustrates a location of reference bits.

FIG. 16 illustrates coefficients used to decide whether to attempt thelook-ahead.

FIG. 17 illustrates post look ahead head bit context model neighborhoodcoefficient contribution.

FIG. 18 illustrates an example of a parent coefficient and bit plane.

FIG. 19 illustrates a neighborhood and parent coefficient contributionto the head bit context model.

FIG. 20 is a flow chart for transform style context model.

FIG. 21 is a flow chart of one embodiment of the decoding process of thepresent invention.

FIG. 22 is a flow chart of one embodiment of the decoding process of thepresent invention.

FIG. 23 illustrates a typical distribution for lossy reconstruction.

FIG. 24 illustrates a method for computing the inverse TT-transform.

FIG. 25 shows the weights used to compute P_(f) across tile boundaries(full-frame).

FIG. 26 shows the weights used to compute P_(t) on a single tileboundary with mirroring.

FIG. 27 illustrates weights for calculating P_(f)-P_(t) approximately.

FIGS. 28 and 28B illustrate deciding linear reconstruction (FIG. 28A) orstep edge reconstruction (FIG. 28B) using five S coefficients.

FIG. 29 illustrates an edge extraction embodiment that generates adifference of gaussian edge extraction for one resolution.

FIG. 30 illustrates a partial transform for use in reconstruction.

FIG. 31 illustrates a clipping reconstruction.

FIG. 32 illustrates the correspondence between the period of a sinusoidand a negative correlation.

FIG. 33 illustrates an example of a single tile buffer.

FIG. 34 illustrates one embodiment of an FSM coder for use with commonoccurrence context modeling.

FIG. 35 illustrates a single lookup table that may be included in an FSMcoder for use with common occurrence context modeling.

FIG. 36 is a flow diagram of one embodiment of the binary-style contextmodel.

FIG. 37 illustrates the neighbor coefficients that contribute to thecontext for the binary-style context model.

FIGS. 38A and 38B illustrate two examples of nine pointers that aremaintained to point to memory locations for the next context.

FIG. 39 illustrates a normalized alignment.

FIG. 40 illustrates a pyramidal alignment.

FIG. 41 illustrates one embodiment of the BVI tag.

DETAILED DESCRIPTION OF THE PRESENT INVENTION

A method and apparatus for compression and decompression are described.In the following description, numerous details are set forth, such asnumbers of bits, bit rates, types of filters, etc. It will be apparent,however, to one skilled in the art, that the present invention may bepracticed without these specific details. In other instances, well-knownstructures and devices are shown in block diagram form, rather than indetail, in order to avoid obscuring the present invention.

Some portions of the detailed descriptions which follow are presented interms of algorithms and symbolic representations of operations on databits within a computer memory. These algorithmic descriptions andrepresentations are the means used by those skilled in the dataprocessing arts to most effectively convey the substance of their workto others skilled in the art. An algorithm is here, and generally,conceived to be a self-consistent sequence of steps leading to a desiredresult. The steps are those requiring physical manipulations of physicalquantities. Usually, though not necessarily, these quantities take theform of electrical or magnetic signals capable of being stored,transferred, combined, compared, and otherwise manipulated. It hasproven convenient at times, principally for reasons of common usage, torefer to these signals as bits, values, elements, symbols, characters,terms, numbers, or the like.

It should be borne in mind, however, that all of these and similar termsare to be associated with the appropriate physical quantities and aremerely convenient labels applied to these quantities. Unlessspecifically stated otherwise as apparent from the followingdiscussions, it is appreciated that throughout the present invention,discussions utilizing terms such as “processing” or “computing” or“calculating” or “determining” or “displaying” or the like, refer to theaction and processes of a computer system, or similar electroniccomputing device, that manipulates and transforms data represented asphysical (electronic) quantities within the computer system's registersand memories into other data similarly represented as physicalquantities within the computer system memories or registers or othersuch information storage, transmission or display devices.

The present invention also relates to apparatus for performing theoperations herein. This apparatus may be specially constructed for therequired purposes, or it may comprise a general purpose computerselectively activated or reconfigured by a computer program stored inthe computer. Such a computer program may be stored in a computerreadable storage medium, such as, but is not limited to, any type ofdisk including floppy disks, optical disks, CD-ROMs, and magneto-opticaldisks, read-only memories (ROMs), random access memories (RAMs), EPROMs,EEPROMs, magnetic or optical cards, or any type of media suitable forstoring electronic instructions, and each coupled to a computer systembus. The algorithms and displays presented herein are not inherentlyrelated to any particular computer or other apparatus. Various generalpurpose machines may be used with programs in accordance with theteachings herein, or it may prove convenient to construct morespecialized apparatus to perform the required method steps. The requiredstructure for a variety of these machines will appear from thedescription below. In addition, the present invention is not describedwith reference to any particular programming language. It will beappreciated that a variety of programming languages may be used toimplement the teachings of the invention as described herein.

The following terms are used in the description that follows. Adefinition has been included for these various terms. However, thedefinition provided should not be considered limiting to the extent thatthe terms are known in the art. These definitions are provided to helpin the understanding of the present invention.

-   alignment: The degree of shifting of the transform coefficients in a    frequency band with respect to the other frequency bands.-   Arithmetic coding: Shannon/Elias Coding with finite precision    arithmetic, not necessarily a binary entropy coder.-   B-coding: A binary entropy coder that uses a finite state machine    for compression. Unlike Huffman coding, using the finite state    machine does well with binary symbols, and is useful for a range of    input probabilities.-   Binary entropy coder: A noiseless coder which acts on binary    (yes/no) decisions, often expressed as the most probable symbol    (mps) and least probable symbol (lps).-   binary-style: Coding style with edge-fill Gray encoding of the    pixels and a particular context model.-   binary-style context A context model for bi-level and limited-level-   model: image data.-   bit-significance: A number representation, similar to sign    magnitude, with head bits, followed by the sign bit, followed by    tail bits, if any. The embedding encodes in bit-plane order with    respect to this representation.-   child-based order: A scan order through a two dimensional image. It    is similar to raster order except that the scan works on two by two    blocks. Consider scanning a “parent” frequency band in raster order.    Each coefficient will have four children. These children are ordered    from top-left, top-right, bottom-left, and bottom-right followed by    the next parent and the next set of four children and so on until    the end of the line. Then processing returns to the next two lines    and eventually ends in the lower right corner. No lines are skipped.    Child-based order is also referred to as 2×2 block order.-   codestream: A code portion of image data, including the header    signaling. In an alternate embodiment, the header signaling is not    included.-   coefficient: Components after the transform.-   components: Constituent parts of the image. The components make up    the pixels. For example, the red, green, and blue bands are    component bands. Each individual pixel is made up of a red, green,    and blue component. Components and component bands can contain any    type of information that has a spatial mapping to the image.-   context model: Causally available information relative to the    current bit to be coded that gives historically-learned information    about the current bit, enabling conditional probability estimation    for entropy coding.-   efficient transform: Transform that achieves the best energy    compaction into the coefficients while using the minimum number of    bits to represent those coefficients.-   Embedded context model: A context model which separates the context    bins and results into levels of importance in such a way that    effective lossy compression is obtained if the more important values    are retained.-   Embedded with ordering: A special case of embedded context models    where there is not an explicit labeling of importance, but rather    the compressed data is ordered with the most important data in the    front.-   embedded quantization: Quantization that is implied by the    codestream. For example, if the importance levels are placed in    order, from the most important to the least, then quantization is    performed by simple truncation of the codestream. The same    functionality is available with tags, markers, pointers, or other    signaling. Multiple quantizations can be performed on an image at    decode, but only one embedded quantization can be performed at    encode time.-   entropy coder: A device that encodes or decodes a current bit based    on a probability estimation. An entropy coder may also be referred    to herein as a multi-context binary entropy coder. The context of    the current bit is some chosen configuration of “nearby” bits and    allows probability estimation for the best representation of the    current bit (or multiple bits). In one embodiment, an entropy coder    may include a binary coder, a parallel run-length coder or a Huffman    coder.-   entry point: A point in the coded data that starts with a known    coding state. The decoder can start decoding at this point without    decoding the previous data. In most cases, this requires that the    context and the binary entropy coder be reset into an initial state.    The coded data for each coding unit begins at an entry point.-   fixed-length: A system that converts a specific block of data to a    specific block of compressed data, e.g., BTC (block truncation    coding) and some forms of VQ (vector quantization). Fixed-length    codes serve fixed-rate and fixed-size applications, but the    rate-distortion performance is often poor compared with    variable-rate systems.-   fixed-rate: An application or system that maintains a certain pixel    rate and has a limited bandwidth channel. In one embodiment, to    attain this goal, local average compression is achieved rather than    a global average compression. For example, MPEG requires a    fixed-rate.-   fixed-size: An application or system that has a limited size buffer.    In one embodiment, to attain this goal, a global average compression    is achieved, e.g., a print buffer. (An application can be    fixed-rate, fixed-size, or both.)-   frequency band: Each frequency band describes a group of    coefficients resulting from the same sequence of filtering    operations.-   head bits: In bit-significance representation, the head bits are the    magnitude bits from the most significant up to and including the    first non-zero bit.-   Huffman Coder: Generally, a fixed length code which produces an    integral number of bits for each symbol.-   idempotent: Coding that enables an image to be decompressed in a    lossy form and recompressed to the same lossy codestream. image    tile: A rectangular region chosen to enable defining a grid of    contiguous non-overlapping sub-images, each with identical    parameters. In one embodiment, the coding operations operate only on    the pixel and coefficient data in one image tile. This restriction    allows random access and region of interest decompression. In one    embodiment, image tiles are all the same size, except possibly for    the right or bottom tiles. In one embodiment, image tiles can be any    size up to and including the whole image.-   importance levels: The unit of coded data which corresponds, before    compression, to an entire bit-plane of the embedded data. The    importance level includes all appropriate bit-planes from the    different coefficient frequency bands.-   LPS (Least Probable Symbol): The outcome in a binary decision with    less than 50% probability. When the two outcomes are equally    probable, it is unimportant which is designated mps or lps as long    as both the encoder and decoder make the same designation.-   Lossless/Noiseless/Reversible coding: Compressing data in a manner    which allows perfect reconstruction of the original data.-   Lossy Coding: Coding of data which does not guarantee perfect    reconstruction of the original data. The changes to the original    data may be performed in such a way as to not be visually    objectionable or detectable. Often fixed rate is possible.-   MPS (Most Probable Symbol): The outcome of a binary decision with    more than 50% probability.-   overlapped transform: A transform where a single source sample point    contributes to multiple coefficients of the same frequency. Examples    include many wavelets and the Lapped Orthogonal Transform.-   parent coefficient: The coefficient or pixel in the next higher    pyramidal level that covers the same image space as the current    coefficient or pixel. For example, the parent of the 1SD    coefficients is the 2SD coefficients which is the parent of the 3SD    coefficients in FIG. 1.-   Probability Estimation Machine/Module: Part of a coding system which    tracks the probability within a context.-   progressive pixel depth: A codestream that is ordered with deepening    bit-planes of data at full image resolution.-   progressive pyramidal: Succession of resolutions where each lower    resolution is a linear factor of two in each dimension (a factor of    four in area).-   pyramidal level: Place in the wavelet decomposition pyramid. This is    directly related to resolution.-   Q-Coder A binary arithmetic coder where additions have been    substituted for multiplications and probabilities limited to    discrete values and probability estimates are updated when bits are    output.-   quantization: This selective elimination, or non-use, of data or    information. Quantization necessarily leads to an imperfect (lossy)    reconstruction. However, intelligent quantization can lead to good    quality for the given amount of data.-   raster order: A scan order through a two dimensional image. It    starts in the upper left corner, moves left to right, then returns    to the left side of the next line, finally ending in the lower right    corner. No lines are skipped.-   reversible transform: In one embodiment, a reversible transform is    an efficient transform implemented with integer arithmetic whose    compressed results can be reconstructed into the original.-   tag: An uncoded part of the codestream that signals information    about the coding characteristics and the bounds of the codestream.    In one embodiment, tags have an identifying number that conveys    their function.-   tail-bits (or tail): In bit-significance representation, the tail    bits are the magnitude bits with less significance than the most    significant non-zero bit.-   tile data segment: Portion of the codestream fully describing one    coding unit.-   Transform style: Coding style that uses a reversible wavelet    transform for energy compaction.-   TS-transform: Two-Six transform, a specific reversible wavelet    filter pair with a 2-tap low pass analysis and a 6tap high pass    analysis filter. The synthesis filters are quadrature mirror of the    analysis filters.-   TT-transform: Two-Ten transform, a specific reversible wavelet    filter pair with a 2-tap low pass analysis and a 10-tap high pass    analysis filter. The synthesis filters are quadrature mirror of the    analysis filters.-   unified lossless/lossy: The same compression system provides a    codestream capable of lossless or lossy reconstruction. In one    embodiment of the present invention, this codestream is capable of    both without settings or instructions to the encoder.-   wavelet filters: The high and low pass synthesis and analysis    filters used in wavelet transform.-   wavelet transform: A transformation with both “frequency” and “time    (or space)” domain constraints. In one embodiment, it is a transform    comprising a high pass filter and a low pass filter. The resulting    coefficients are decimated by two (critically filtered) and the    filters are applied to the low pass coefficients.

wavelet trees: The coefficients, and the pixels, that are related to asingle coefficient in the SS section of the highest level waveletdecomposition. The number of coefficients is a function of the number oflevels. FIG. 1 illustrates the coefficients included in a wavelet tree.The span of a wavelet tree is dependent on the number of decompositionlevels. For example, with one level of decomposition, a wavelet treespans four pixels, with two levels it spans 16, etc. Table 1 belowillustrates the number of pixels affected by a wavelet tree fordifferent levels. In two dimensions, each wavelet tree comprises threesubtrees called SD, DD and DS. TABLE 1 Span of a Wavelet Tree forDifferent Levels of Decompression Width Height Total 1 level 2 2 4 2levels 4 4 16 3 levels 8 8 64 4 levels 16 16 256 5 levels 32 32 1024 6levels 64 64 4096Overview of the Present Invention

The present invention provides a compression/decompression system havingan encoding portion and a decoding portion. The encoding portion isresponsible for encoding input data to create compressed data, while thedecoding portion is responsible for decoding previously encoded data toproduce a reconstructed version of the original input data. The inputdata may comprise a variety of data types, such as image (still orvideo), audio, etc. In one embodiment, the data is digital signal data;however, analog data digitized, text data formats, and other formats arepossible. The source of the data may be a memory or channel for theencoding portion and/or the decoding portion.

In the present invention, elements of the encoding portion and/or thedecoding portion may be implemented in hardware or software, such asthat used on a computer system. The present invention provides alossless compression/decompression system. The present invention mayalso be configured to perform lossy compression/decompression. Thepresent invention may be configured to perform parsing of compresseddata without decompressing.

Overview of the System of the Present Invention

The present invention represents the smooth edges and flat regions foundin natural images quite well. Using reversible embedded wavelets, thepresent invention compresses deep pixel images. However, reversibleembedded wavelets, and other wavelet and sinusoidal transform systems,are not good at representing sharp edges found in text or graphicimages. This type of image can be compressed well by Gray codingfollowed by context-based bit-plane encoding, like the JBIG.Furthermore, noise free computer-generated images are well-modeled bybinary style.

The present invention provides a binary style for compression of binaryand graphic images. This also improves compression on some images thatdo not use the full dynamic range. In the binary style, the presentinvention encodes bit-planes of the image without using the wavelettransform.

FIG. 2 is a block diagram of one embodiment of a compression system ofthe present invention that employs the binary style. Note the decodingportion of the system operates in reverse order, along with the dataflow. Referring to FIG. 2, an input image 201 into a multi-componenthandling mechanism 211. The multi-component handling mechanism 211provides optional color space conversion and optional handling ofsubsampled image components. Style select mechanism 210 determineswhether the image is a continuous-tone image or a binary image, or whichportions of an image have such characteristics. The image data isforwarded onto the style select mechanism 210 which sends the image dataor portions of the image data to either the wavelet style processing(blocks 202, 203, 205) or the binary style processing (block 204). Inthe present invention, the decision as to which mode to use is datadependent. In one embodiment, the style select mechanism 210 comprises amultiplexer. Style select 210 is not used during decoding.

In the wavelet style, the reversible wavelets block 202 performs areversible wavelet transform. The output of block 202 is a series ofcoefficients. The embedded order quantization block 203 places thecoefficients in bit-significance representation and then labels thecoefficients in order to create an alignment of all of the coefficientsin input image 201 (as generated by reversible wavelet block 202).

The image data 201 is received and (after optimal multicomponenthandling) transformed using reversible wavelets in wavelet transformblock 202, as defined below, to produce a series of coefficientsrepresenting a multi-resolution decomposition of the image. Thereversible wavelet transforms of the present invention are notcomputationally complicated. The transforms may be performed in softwareor hardware with no systematic error. Furthermore, the wavelets of thepresent invention are excellent for energy compaction and compressionperformance. These coefficients are received by the embedded orderquantization block 203.

The embedded order quantization block 203 provides embedded orderquantization, as described below. The result is an embedded data stream.The embedded data stream allows a resulting codestream to be quantizedat encode time, transmission time, or decode time. In one embodiment,embedded order quantization block 203 orders and converts thecoefficients into sign-magnitude format.

The embedded data stream is received by the context model 205, whichmodels data in the embedded data stream based on their significance (asdescribed below later). In the case of the transform mode, the“bit-planes” are importance level planes of the transform coefficientsand context model 205 conditions wavelet coefficients inbit-significance representation.

The results of ordering and modeling comprise decisions (or symbols) tobe coded by the entropy coder 206. In one embodiment, all decisions aresent to a single coder. In another embodiment, decisions are labeled bysignificance, and decisions for each significance level are processed bydifferent (physical or virtual) multiple coders. The bit stream(s) areencoded in order of significance using entropy coder 206. In oneembodiment, entropy coder 206 comprises one or more binary entropycoders. In another embodiment, Huffman coding is used.

In the binary style, Gray coding block 204 performs Gray coding on thepixels in input image 201. Gray coding is a pixel operation that takesadvantage of some of the correlation between the bit-planes of thepixels. This is because for any value of x and x+1, the gray (x) andgray (x+1) differ by only one bit in their radix 2 representations. Inone embodiment, gray coding block 204 performs a point wise transform on8 bit pixels:gray (x)=x XOR x/2The present invention is not limited to this form of Gray coding, nor islimited to using pixels that are 8-bits in size. Note, however, thatemploying the above equation has an advantage of allowing a pixel to bereconstructed with only some of the most significant bits available, asis the case in progressive-by-bit-plane transmission. In other words,this form of Gray coding preserves the bit-significance ordering.

In the binary style, the data is encoded by bit-plane using a contextmodel in coding block 204 and coder 206. In one embodiment, contextmodel in coding block 204 conditions the current bit using spatial andimportance level information.

With the binary style, a JBIG-like context model is used on Gray codedpixels. In one embodiment, each bit-plane of the image tile is codedseparately with each individual bit being conditioned and coded inraster order using the values of ten surrounding bits. FIG. 3illustrates the geometric relationship of the context model for each bitof each bit-plane in the binary style. The conditioning bits lead to anadaptive probability estimate for each unique pattern. Note that somedifferent templates may be used for the context model of the binaryentropy coder when used in the bit-plane entropy coding of the Graycoded values. FIG. 4 illustrates seven pixels and two bits of bit planeinformation for 2⁹ context bins.

Using this context and the value of the current bit, binary coder 206creates a bit stream. The same binary entropy coder 206 is used to codedata from both the transform mode and the binary style. In oneembodiment, binary coder 206 comprises a finite state machine coder thatis implemented with a look-up table. Note that the present invention maybe used with any binary entropy coder, such as the Q-coder, QM-coder ora high speed parallel coder.

Because the binary coder 206 is the same for either style and the Graycoding and the binary context model are simple, very little extraresources are required to have the binary style and transform style inthe same system. Furthermore, while the context model configuration isdifferent, the resource requirements are the same for both modes. Thatis, both use the same memory for storing contexts and both use the samebinary entropy coder.

The present invention may be performed on the entire image, or, morecommonly, on tiled segments of the image. Some tiles may be bettercompressed with the transform style and others with the binary style.There are any number of algorithms possible for choosing which mode touse. If tiles are used, then random access on a tile basis is possible.Also, regions of interest can be decoded separately to a higherfidelity. Finally, the choice of whether to use the transform or binarystyle can be decided on a tile-by-tile basis.

Also note that the image is still progressive by bit-plane using thedual mode system of the present invention and may be encoded in ahierarchical format as taught by JBIG.

With respect to decoding, one bit in the header of the tile may be usedto denote the style used to encode the data. Style select 210 is notused. A lossless mapping, if possible, from the original dynamic rangeto a lower dynamic range, such as by histogram compaction (describedbelow) can help further. A look ahead, such as in JBIG, may be used. Thelookahead may employ typical prediction or deterministic prediction,such as in JBIG.

Selection of Binary or Transform Style

Style select 210 selects between the binary style and transform style.In one embodiment, the input image is encoded with both styles and styleselect 210 selects the style which produces the lower bit rate (assuminglossless compression). In other words, which ever mode compresses thebest is selected. This method does not have as high a cost as might beexpected since both the binary style and transform mode are relativelyquick in software and small in hardware. A derivative of this method isto bypass the coder and use entropy values for determining the lower bitrate.

In an alternate embodiment, the present invention creates a complete (orpartial) histogram of the pixel values of the image or a histogram ofthe differences between pairs of adjacent pixel values. In the case ofthe histogram of pixel values, statistical analysis of this data, suchas if the histogram is peaked at a few values, far fewer than thedynamic range of the pixel depth, then the binary style is used.

In one embodiment, the present invention creates a complete (or partial)histogram of the first order differences between pairs of adjacentpixels. For a normal image, such a histogram is very Laplacian andwavelet style would be used. However, if this histogram is not peakedwith a Laplacian distribution, then the binary style is used.

Both types of histograms may be generated and used together to selectthe style.

The d_(n) filter output of the TS-transform or the TT-transform, both ofwhich are discussed later, is similar to the first order statistics.This suggests a method where the transform is performed and thehistogram generated. Based on the histogram, the style is chosen. If itis the transform mode, the system proceeds with the transformcoefficients already generated. If the binary style is chosen thetransform coefficients are discarded (or inverse transformed dependingon whether the pixels were saved) and the system proceeds with thebinary style.

In another embodiment, segmentation and/or previous knowledge of thedocument types may help determine which styles to select.

In some embodiments, the tiling size is chosen to maximize the benefitof the two styles.

Note that in one embodiment, the system of the present invention doesnot include binary style coding and, thus, only uses the reversibleembedded wavelet compression (CREW) and decompression only.

Furthermore, the present invention provides for a decompression system.The decompression system of the present invention includes components toreverse those operations performed by the compression system, whether ina lossless or lossy manner. To avoid obscuring the present invention,the decompression system is only described where it differs from thecompression system (other than simply reversing the dataflow).

The Codestream of the Present Invention

In the image compression system of the present invention, a digitalimage is divided into rectangular regions called tiles which are codedindependently. Further, a digital image is composed of multiplecomponents, each describing different aspects of a pixel, or pixels.(The most common example are color components, where each componentdescribes the amount of a particular color, like red, green, or blue.)These image components are also coded independently.

Image tiles are complete, independently-coded sub-images of the wholeimage, defined by a regular rectangular grid placed on the image andnumbered in raster order, as in FIG. 5. The tiles are usually ordered inthe codestream also in raster order. The tiles on the right and bottomcan be different sizes depending on the original image and the tilesize. (No extra coding is required for these odd-sized tiles.) The sizeof the tiles are user-definable at encode time and can be any height andwidth, up to the size of the image.

The choice of tile size has a major impact on performance. Small tiles,especially in the vertical dimension on raster-ordered images, can allowthe use of less workspace memory. However, if the tile size is toosmall, compression efficiency is reduced by three factors: the signalingoverhead, the loss of coding efficiency on the boundaries of the tile,and the start-up adaptation of the entropy coder. It is beneficial tohave tile dimensions that are a multiple of the extent of a lowestfrequency component, which is a function of the number of levels(²number-of-levels). Tiles of 128 by 128 or 256 by 256 seem reasonablein many applications, depending on the size of the original image.

There is nothing in the definition or syntax of tiles that prohibitscompressing a sequence of images. Thus, tiled images could be differentimages in time (like a movie) or in space (like 3D cross sections likeMRI).

Each tile contains one or more components. Each component covers, orspans, the entire tile, although each can be of different resolution. inone embodiment, every tile has at least one sample from each component.These components can be of different pixel depths and can be coded indifferent styles. Each component is coded independently, but the codeddata can be interleaved on an importance level basis.

Headers and Tags

The present invention uses tags to delimit and signal thecharacteristics of the codestream. Every codestream has at least twoheaders: the main header at the beginning of the image and a tile headerat the beginning of each tile. (Every codestream contains at least onetile.)

In one embodiment, five kinds of tags are used: delimiting, fixedinformation, functional, pointer, and informational tags. Delimitingtags are used to frame the headers and the data. Fixed information tagsprofile required information about an image. Functional describe thecoding functions used to code the entire title or image. Pointer tagspoint to the importance level in a tile (or to the next tile).Informational tags provide optional information about the data.

Pointer Tags

Pointer tags either provide a length or point into the codestream.Pointer tags may appear in the main header or in all of the tileheaders. The IEM tag or IET tag described below are used if there is anentry point in the codestream or if there are multiple components. (Anentry point is required whenever the component being coded changes. Itis also required if the first component in a multi-component tile is notcomponent 0.) The ILM and ILT tags are optional; they point to the startof the coded data in an importance level.

The presence of any of these tags in the main header indicates that allthese tags are in the main header. When none of these tags is in themain header, then all of the tags will be in the tile headers. Theadvantage to having the pointer tags all occur in the main header isthat the decoder or parser can select and/or quantize a codestreamwithout rewinding. This ability could be advantageous for applicationsthat require fast or limited decoders and/or parsers. If the encoder isnot capable of rewinding all the way to the main header, then thepointer tags can be distributed in the tile headers. This distributionis advantageous when encoding very large images or when using a hardwareencoder where rewinding or storing the codestream is difficult.

Importance Level Entry Points, Main Header (IEM)

The IEM tag comprises a list of pointers to all the entry points inevery tile for a given component. Each IEM tag is for a differentcomponent. Entry points are found on 8-bit boundaries at the beginningof an importance level. The importance levels that have entry points areselected at encode time. However, an entry point is required every timethe component in the data stream changes. Also, if the first componentis not component number 0, an IEM pointer entry with a pointer value, 0,is required. At each entry point, the entropy coder is reset. Therefore,these entry points are handled by the decoder.

Although every IEM pointer entry must be correct, there can be redundantentries (copies). The entries must be sorted in increasing order of thepointer length. Note that since the components can be interleaved byimportance level, the IEM tags for the different components could beinterleaved.

In one embodiment, the IEM tag is used in the main header, if there areentry points in the file, unless a IET tag (described later) is used inevery tile header. The IEM tag appears once per component, exceptpossibly for component 0. If the only entry point for component 0 isPiem=0, in all tiles, no tag is required for component 0.

The length is variable depending on the number of tiles in the image andthe number of entry points in each tile.

FIG. 6 illustrates the Importance level entry points, main header syntaxof IEM tags with the fields described below: IEM: Marker. Liem: Lengthof tag in bytes, not including the marker. Ciem: Component to which thistag applies. Components are numbered 0, 1, 2, etc. Niemi: Number ofentry points in the ith tile. There is an Niem for each tile in theimage, even if there are no entry points in that tile. Iiemij: Number ofthe importance level starting at the jth entry point ith tile for theCiem component. This Iiem tag and the corresponding Piem tag form a typeof record repeated for each entry point. These records must be in orderfrom the highest to the lowest importance levels that has an entrypoint. Piemij: Number of bytes from the end of the tile header or thebeginning of all coded data in that tile to the jth entry point byte.These records must be in order from the smallest pointer to the largest.res: A filler byte of zeros that is placed at the end, as needed.

Importance Level Entry Points, Tile Header (IET)

The IET tag is a list of pointers to all the entry points in this tilefor a given component. Each IET tag is for a different component. Theseentry points are found on 8-bit boundaries at the beginning of animportance level. The importance levels that have entry points areselected by the encoder. However, an entry point is required every timethe component in the data stream changes. Also, if the first componentis not component number 0, an entry point with a pointer value, 0, isrequired. At each entry point, the entropy coder is reset to a knownstate. Therefore, these entry points must be handled by the decoder.

Although every IET pointer entry is correct, there can be redundantentries (copies). The entries are sorted in increasing order of thepointer length. Note that since the components can be interleaved byimportance level, the IET tags for the different components could beinterleaved.

The IET tag is used in the every tile header for a tile with entrypoints unless a IEM tag is used in the main header, and appears once percomponent, except possibly for component 0. If the only entry point forcomponent 0 is Piem=0, no tag is required for component 0.

The length of the IET tag is variable depending on the number of entrypoints in each tile.

FIG. 7 illustrates the Importance level entry points, tile header syntaxof IET tags, which include the following fields. IET: Marker. Liet:Length of tag in bytes, not including the marker. Ciet: Component towhich this tag applies. Components are numbered 0, 1, 2, etc. Iieti:Number of the importance level starting at the ith entry point for theCiet component. This Iiet tag and the corresponding Piet tag form a typeof record repeated for each entry point. These records are in order fromthe highest to the lowest importance levels that has an entry point.Pieti: Number of bytes from the end of the tile header or the beginningof all coded data in that tile to the ith entry point byte. Theserecords are in order from the smallest pointer to the largest. res: Afiller byte of zeros that is placed at the end.

Importance Level Locators, Main Header (ILM)

The ILM tag is a list of pointers that point to encoder selectedimportance levels in every tile for a given component. These importancelevels are not necessarily found on 8-bit boundaries. Optimally, thepointer points to the first byte that contains data for the importancelevel being located (and no data from any previous importance level).However, the pointer can point to any data byte that contains data forthat importance level.

Although every ILM pointer entry is correct, there can be redundantentries (copies). The entries are sorted in increasing order of thepointer length. Note that since the components can be interleaved byimportance level using entry points, the ILM tags for the differentcomponents could be interleaved.

The ILM tag is optional in the main header but may not be used ifpointer tags are in tile headers. There is up to one ILM per componentin the main header.

The length of the ILM tag is variable depending on the number of tilesin the image and the number of locator points in each tile.

FIG. 8 illustrates the importance level locators, main header syntax forthe ILM tag, which include the following fields. ILM: Marker. Lilm:Length of tag in bytes, not including the marker. Cilm: Component towhich this tag applies. Nilmi: Number of locators in the ith tile. Thereis an Niem for each tile in the image, even if there are no locators inthat tile. Iilmij: Number of the jth importance level starting in theith tile for the Cilm component. This Iilm tag and the correspondingPilm tag form a type of record repeated for each locator. These recordsmust be in order from the highest to the lowest importance levels.Pilmij: Number of bytes from the end of the tile header or the beginningof all coded data in that tile to a data byte containing data from thejth importance level. These records are in order from the smallestpointer to the largest. res: A filler byte of zeros that is placed atthe end, as needed.

Importance Level Locators, Tile Header (ILT)

The ILT tag is a list of pointers that point to encoder selectedimportance levels in every tile for a given component These importancelevels are not necessarily found on 8-bit boundaries. Optimally, thepointer points to the first byte that contains data for the importancelevel being located (and no data from any previous importance level).However, the pointer can point to any data byte that contains data forthat importance level.

Although every ILT pointer entry is correct, there can be redundantentries (copies). The entries are sorted in increasing order of thepointer length. Note that since the components can be interleaved byimportance level using entry points, the ILT tags for the differentcomponents could be interleaved.

The IET tag is up to one ILT per component in the tile headers but maynot be used if pointer tags are in the main header. The length of theIET tag is variable depending on the number of locator points in eachtile.

FIG. 9 illustrates the importance level locators, tile header syntax ofthe ILT tag, which include the following fields: ILT: Marker. Lilt:Length of tag in bytes, not including the marker. Cilt: Component towhich this tag applies. Components are numbered 0, 1, 2, etc. Iilti:Number of the importance level starting at the ith entry point for theCilt component. This Iilt tag and the corresponding Pilt tag form a typeof record repeated for each locator. These records are in order from thehighest to the lowest importance levels that has a locator. Pilmi:Number of bytes from the end of the tile header or the beginning of allcoded data in that tile to a data byte containing data from the jthimportance level. These records are in order from the smallest pointerto the largest. res: A filler byte of zeros that is placed at the end.

Informational Tags

Information tags are strictly information and are not necessary for adecoder. However, these tags might assist a parser or decoder. The Bitsversus importance levels (BVI) tag is an example of an informationaltag, and is shown in FIG. 41 and is described in greater detail below.

Reversible Wavelets

The present invention employs compression by reversible wavelets.

Wavelet Decomposition

The present invention initially performs decomposition of an image (inthe form of image data) or another data signal using reversiblewavelets. In the present invention, a reversible wavelet transformcomprises an implementation of an exact-reconstruction system in integerarithmetic, such that a signal with integer coefficients can belosslessly recovered. An efficient reversible transform is one withtransform matrix of determinant equals 1 (or almost 1).

By using reversible wavelets, the present invention is able to providelossless compression with finite precision arithmetic. The resultsgenerated by applying the reversible wavelet transform to the image dataare a series of coefficients.

The reversible wavelet transform of the present invention may beimplemented using a set of filters. In one embodiment, the filters are aTwo-tap low-pass filter and a Six-tap high-pass filter to implement atransform referred to herein as the TS transform, or 2,6 transform. Inanother embodiment, the filters are a Two-tap low-pass filter and aTen-tap high-pass filter to implement a transform referred to herein asthe TT transform, or 2,10 transform.

Two-Dimensional Wavelet Decomposition

Using the low-pass and high-pass filters of the present invention, amulti-resolution decomposition is performed. The number of levels ofcomposition is variable and may be any number; however, currently thenumber of decomposition levels equals from two to eight levels. Themaximum number of levels is the log₂ of the maximum of the length orwidth of the input.

The most common way to perform the transform on two-dimensional data,such as an image, is to apply the one-dimensional filters separately,i.e., along the rows and then along the columns. The first level ofdecomposition leads to four different bands of coefficients, referred toherein as SS, DS, SD, and DD. The letters refer to the smooth (S) anddetail (D) filters defined above, which correspond to low (L) and high(H) pass filters respectively. Hence, the SS band consist ofcoefficients from the smooth filter in both row and column directions.

Each frequency subband in a wavelet decomposition can be furtherdecomposed. The most common practice is to only decompose the SSfrequency subband further, and may include further decomposing of the SSfrequency subband in each decomposition level as each is generated. Sucha multiple decomposition is referred to as a pyramidal decomposition.The designations SS, SD, DS, DD and the decomposition level numberdenote each decomposition.

Note that with either the TS or TT transforms of the present invention,the pyramidal decomposition does not increase the coefficient size.

If the reversible wavelet transform is recursively applied to an image,the first level of decomposition operates on the finest detail, orresolution. At a first decomposition level, the image is decomposed intofour sub-images (e.g., subbands). Each subband represents a band ofspatial frequencies. The first level subbands are designated 1SS, 1SD,1DS, and 1DD. The process of decomposing the original image involvessubsampling by two in both horizontal and vertical dimensions, such thatthe first level subbands 1SS, 1SD, 1DS and 1DD each have one-fourth asmany coefficients as the input has pixels (or coefficients) of theimage.

Subband 1SS contains simultaneously low frequency horizontal and lowfrequency vertical information. Typically a large portion of the imageenergy is concentrated in this subband. Subband 1SD contains lowfrequency horizontal and high frequency vertical information (e.g.,horizontal edge information). Subband 1DS contains high frequencyhorizontal information and low frequency vertical information (e.g.,vertical edge information). Subband 1DD contains high frequencyhorizontal information and high frequency vertical information (e.g.,texture or diagonal edge information).

Each of the succeeding second, third and fourth lower decompositionlevels is produced by decomposing the low frequency SS subband of thepreceding level. This subband 1SS of the first level is decomposed toproduce subbands 2SS, 2SD, 2DS and 2DD of the moderate detail secondlevel. Similarly, subband 2SS is decomposed to produce coarse detailsubbands 3SS, 3SD, 3DS and 3DD of the third level. Also, subband SS₂ isdecomposed to produce coarser detail subbands 4SS, 4SD, 4DS and 4DD ofthe third level. Due to subsampling by two, each second level subband isone-sixteenth the size of the original image. Each sample (e.g., pixel)at this level represents moderate detail in the original image at thesame location. Similarly, each third level subband is 1/64 the size ofthe original image. Each pixel at this level corresponds to relativelycoarse detail in the original image at the same location. Also, eachfourth level subband is 1/256 the size of the original image.

Since the decomposed images are physically smaller than the originalimage due to subsampling, the same memory used to store the originalimage can be used to store all of the decomposed subbands. In otherwords, the original image and decomposed subbands 1SS and 2SS arediscarded and are not stored in a three level decomposition.

Although only four subband decomposition levels are described,additional levels could be developed in accordance with the requirementsof a particular system. Also, with other transformations such as DCT orlinearly spaced subbands, different parent-child relationships may bedefined.

Note that pyramidal decomposition does not increase the coefficient sizewith the wavelet filters of the present invention.

In other embodiments, other subbands in addition to the SS may bedecomposed also.

Tree Structure of Wavelets

There is a natural and useful tree structure to wavelet coefficients ina pyramidal decomposition. A result of the subband decomposition is asingle SS frequency subband corresponding to the last level ofdecomposition. On the other hand, there are as many SD, DS, and DD bandsas the number of levels. The tree structure defines the parent of acoefficient in a frequency band to be a coefficient in a same frequencyband at a lower resolution and related to the same spatial locality.

In the present invention, each tree comprises the SS coefficients andthree subtrees, namely the DS, SD and DD subtrees. The processing of thepresent invention is typically performed on the three subtrees. The rootof each tree is a purely smooth coefficient. For a two-dimensionalsignal such as an image, there are three subtrees, each with fourchildren. The tree hierarchically is not limited to two dimensionalsignals. For example, for a one dimensional signal, each subtree has onechild. Higher dimensions follow from the one-dimensional andtwo-dimensional cases.

The process of multi-resolution decomposition may be performed using afiltering system. For examples of a two-dimensional, two-leveltransform, a two-dimensional, two-level transform implemented usingone-dimensional exemplary filters, see U.S. patent application Ser. No.08/498,695, filed Jun. 30, 1995 and entitled “Method and Apparatus ForCompression Using Reversible Wavelet Transforms and an EmbeddedCodestream” and U.S. patent application Ser. No. 08/498,036, filed Jun.30, 1995, entitled “Reversible Wavelet Transform and Embedded CodestreamManipulation”.

Performing the Forward Wavelet Transform

In the present invention, the wavelet transform is performed with two1-D operations, horizontal then vertical. In one embodiment, one pieceof hardware performs the horizontal operation while another performs thevertical operations.

The number of levels determine the number of iterations. In oneembodiment, a four level decomposition is performed using the TTtransform in both the horizontal and vertical directions. In anotherembodiment, a four level decomposition is performed using fourTS-transforms instead.

The transform of the present invention is computationally efficient. Inone embodiment, the present invention orders the computations performedby the transform to reduce the amount of both on-chip and off-chipmemory and bandwidth required.

Computation for One Wavelet Tree

The following equations define both the TS-transform and theTT-transform. For an input x(n), the output of the low pass filter, thesmooth signal s(n), and the high pass filter, the detail signal d(n) arecomputed as shown in the equation below. $\left( \begin{matrix}{{s(n)} = \left\lfloor \frac{{x\left( {2n} \right)} + {x\left( {{2n} + 1} \right)}}{2} \right\rfloor} \\{{d(n)} = {{x\left( {2n} \right)} - {x\left( {{2n} + 1} \right)} + {t(n)}}}\end{matrix}\quad \right.$The inverse transform is shown in the equation below.$\left( \begin{matrix}{{x\left( {2n} \right)} = {{s(n)} + \left\lfloor \frac{{p(n)} + 1}{2} \right\rfloor}} \\{{x\left( {{2n} + 1} \right)} = {{s(n)} - \left\lfloor \frac{p(n)}{2} \right\rfloor}}\end{matrix}\quad \right.$where p(n) is computed by:p(n)=d(n)−t(n).The TS-transform and the TT-transform differ in the definition of t(n).For the TS-transform.${t(n)} = {\left\lfloor \frac{{- {s\left( {n - 1} \right)}} + {s\left( {n + 1} \right)} + 2}{4} \right\rfloor.}$For the TT-transform,${t(n)} = \left\lfloor \frac{{3{s\left( {n - 2} \right)}} - {22{s\left( {n - 1} \right)}} + {22{s\left( {n + 1} \right)}} - {3{s\left( {n + 2} \right)}} + 32}{64} \right\rfloor$

Note that in the following discussion the notation └.┘ means to rounddown or truncate and is sometimes referred to as the floor function.

These filters may be implemented using only addition and subtractionoperations (plus hardwired bit shifting). For instance, multiplicationby 3 and 22 may be performed by shifts and adds.

Note that in both the reversible TS-transform and TT transform, like theS-transform, the low-pass filter is implemented so that the range of theinput signal x(n) is the same as the output signal s(n). That is, thereis no growth in the smooth output. If the input signal is b bits deep,then the smooth output is also b bits. For example, if the signal is an8-bit image, the output of the low-pass filter is also 8 bits. This isan important property for a pyramidal system where the smooth output isdecompressed further by, for example, successively applying the low-passfilter. In prior art systems, the range of the output signal is greaterthan that of the input signal, thereby making successive applications ofthe filter difficult. Also, there is no systemic error due to roundingin the integer implementation of the transform, so all error in a lossysystem can be controlled by quantization. In addition, the low-passfilter has only two taps which makes it a non-overlapping filter. Thisproperty is important for the hardware implementation.

Embedded Ordering

In the present invention, the coefficients generated as a result of thewavelet decomposition are entropy coded. In the present invention, thecoefficients initially undergo embedded ordering in which thecoefficients are ordered in a visually significant order or, moregenerally, ordered with respect to some error metric (e.g., distortionmetric). Error or distortion metrics include, for example, peak errorand mean squared error (MSE). Additionally, ordering can be performed togive preference to bit-significance spatial location, relevance fordatabase querying, and directionality (vertical, horizontal, diagonal,etc.).

The ordering of the data is performed to create the embeddedquantization of the codestream. In the present invention, two orderingsystems are used: a first for ordering the coefficients and a second forordering the binary values within a coefficient. The ordering of thepresent invention produces a bitstream that is thereafter coded with abinary entropy coder.

Bit-Significance Representation

Most transform coefficients are signed numbers even when the originalcomponents are unsigned (any coefficients output from at least onedetail filter are signed). In one embodiment, the embedded order usedfor binary values within a coefficient is by bit-plane. The coefficientsare expressed in bit-significance representation prior to coding.Bit-significance is a sign-magnitude representation where the sign bit,rather than being the most significant bit (MSB), is encoded with thefirst non-zero magnitude bit. That is, the sign bit follows the firstnon-zero magnitude bit rather than preceding all of the magnitude bits.Also, the sign bit is considered to be in the same bit-plane as the mostsignificant non-zero magnitude bit.

Bit-significance format represents a number using three sets of bits:head, tail, and sign. The head bits are all the zero bits from the MSBup to and including the first non-zero magnitude bit. The bit-plane inwhich the first non-zero magnitude bit occurs defines the significanceof the coefficient. The set of tail bits comprises the magnitude bitsafter the first non-zero magnitude bit to the LSB. The sign bit simplydenotes the sign, where a 0 may represent a positive sign and 1 mayrepresent a negative sign. A number, such as ±2^(n), with a non-zero bitas the MSB has only one head bit. A zero coefficient has no tail or signbits. FIG. 12 illustrates examples of bit-significance representation.Table 2 shows all possible values for form bit coefficients ranging from−7 to 8. TABLE 2 Bit Significance Representation for 4 Bit Values 2′sSign Decimal Complement Magnitude Bit-Significance −8 1000 −7 1001 111111 1 1 −6 1010 1110 11 1 0 −5 1011 1101 11 0 1 −4 1100 1100 11 0 0 −31101 1011 0 11 1 −2 1110 1010 0 11 0 −1 1111 1001 0 0 11 0 0000 0000 0 00 1 0001 0001 0 0 10 2 0010 0010 0 10 0 3 0011 0011 0 10 1 4 0100 010010 0 0 5 0101 0101 10 0 1 6 0110 0110 10 1 0 7 0111 0111 10 1 1

In Table 2, the bit significance representation shown in each columnincludes one or two bits. In the case of two bits, the first bit is thefirst one bit and is followed by the sign bit.

In the case where the values are non-negative integers, such as occurswith respect to the intensity of pixels, the order that may be used isthe bitplane order (e.g., from the most significant to the leastsignificant bitplane). In embodiments where two's complement negativeintegers are also allowed, the embedded order of the sign bit is thesame as the first non-zero bit of the absolute value of the integer.Therefore, the sign bit is not considered until a non-zero bit is coded.For example, using sign magnitude notation, the 16-bit number −7 is:

-   -   1000000000000111        On a bit-plane basis, the first twelve decisions will be        “insignificant” or zero. The first 1-bit occurs at the        thirteenth decision. Next, the sign bit (“negative”) will be        coded. After the sign bit is coded, the tail bits are processed.        The fifteenth and sixteenth decisions are both “1”.

Since the coefficients are coded from most significant bitplane to leastsignificant bitplane, the number of bitplanes in the data must bedetermined. In the present invention, this is accomplished by finding anupper bound on the magnitudes of the coefficient values calculated fromthe data or derived from the depth of the image and the filtercoefficients. For example, if the upper bound is 149, then there are 8bits of significance or 8 bitplanes. For speed in software, bitplanecoding may not be used. In an alternate embodiment, a bitplane is codedonly when a coefficient becomes significant as a binary number.

Coefficient Alignment

The present invention aligns coefficients with respect to each otherbefore the bit-plane encoding. This is because the coefficients in thedifferent frequency subbands represent different frequencies similar tothe FFT or the DCT. By aligning coefficients, the present inventioncontrols quantization. The less heavily quantized coefficients will bealigned toward the earlier bit-planes (e.g., shifted to the left). Thus,if the stream is truncated, these coefficients will have more bitsdefining them than the more heavily quantized coefficients.

FIGS. 39 and 40 illustrate a normalized alignment and a pyramidalalignment, respectively.

In one embodiment, the coefficients are aligned for the bestrate-distortion performance in terms of SNR or MSE. There are manypossible alignments including one that is near-optimal in terms ofstatistical error metrics such as MSE. Alternately, the alignment couldallow a physcovisual quantization of the coefficient data. The alignmenthas significant impact on the evolution of the image quality (or inother words on the rate-distortion curve), but has negligible impact onthe final compression ratio of the lossless system. Other alignmentscould correspond to specific coefficient quantization, Region ofInterest fidelity encoding, or resolution progressive alignment.

The alignment may be signaled in the header of the compressed data or itmay be fixed for a particular application or it may be fixed for aparticular application (i.e., the system only has one alignment). Thealignment of the different sized coefficients is known to both the coderand decoder and has no impact on the entropy coder efficiency.

The bit depths of the various coefficients in a two-level TS-transformand TT-transform decomposition from an input image with b bits per pixelare shown in FIG. 10. FIG. 11 is one embodiment of the multipliers forthe frequency band used for coefficient alignment in the presentinvention. To align the coefficients, the 1-DD coefficient size is usedas a reference, and shifts are given with respect to this size. A shiftof n is a multiplication by 2^(n).

In one embodiment, the coefficients are shifted with respect to themagnitude of the largest coefficient to create an alignment of all thecoefficients in the image. The aligned coefficients are then handled inbit-planes called importance levels, from the most significantimportance level to the least significant importance level. The sign isencoded with the last head bit of each coefficient. The sign bit is inwhatever importance level the last head bit is in. It is important tonote that the alignment simply controls the order the bits are sent tothe entropy coder. Actual padding, shifting, storage, or coding of extrazero bits is not performed.

Table 3 illustrates one embodiment of alignment numbers for aligningcoefficients for normalized alignment, such as shown in FIG. 39. TABLE 3Coefficient Alignment for Normalized Alignment 1-DD 1-DS, 1-SD 2-DD2-DS, 2-SD 3-DD 3-DS, 3-SD 4-DD 4-DS, 4-SD reference Left 1 Left 1 Left2 Left 2 Left 3 Left 3 Left 4

The alignment of different sized coefficients is known to both the coderand the decoder and has no impact on the entropy coder efficiency.

Note that coding units of the same data set may have differentalignments.

Ordering of the Codestream and the Context Model

In one embodiment, the codestream of the present invention comprises ofa main header, tile headers, and tile data. Each tile has at least onedata point from each component in the image. The component data istransformed (using transform-style or binary-style coding) and thenaligned into the importance levels. The importance levels are thenentropy coded using the appropriate context model and the code (e.g.,FSM coder).

Note that in one embodiment the modeling and coding is a straightthrough process. No rearranging of the data is performed. Therefore, theentropy coded importance levels are the minimum possible coded unit inthe data stream. If entry points are placed in front of an importancelevel (or the data is fixed-length) then the code stream can be decodedstarting at that point.

The SS coefficients are classified as the most significant importancelevel. (Note that if the binary-style coding is used and the componentis decomposed zero levels, then the coefficients are considered to be inthe DD frequency band.) These coefficients (whether transform-style orbinary-style) are not entropy coded. The SS coefficients are packed intothe codestream in raster order with the Most Significant Bit (MSB) firstand Least Significant Bit (LSB) last regardless of the depth of thecoefficient. Signed components are stored as unsigned values offset by2ˆ(depth-1). For example 8 bit deep signed components taking on valuesfrom −128 to 127 have 128 added to there value and are stored unsignedfrom 0-255.

In one embodiment for each component the importance levels are orderedin the codestream from most significant (highest number) to leastsignificant.

It is possible to enter and decode at a particular importance level withthe use of entry points. Entry points are pointed to with the IEM or IETtags. The entropy coder can be reset at certain points in thecodestream; the points are decided at encode-time and can only occur atthe beginning of an importance level. This reset creates an entry pointwhere the coding state information (context and probabilities) is aknown initial state. The codestream is padded with bits to the nextmultiple of 8 bits.

The context model determines both the order in which data is coded andthe conditioning used for specific bits of the data. Ordering will beconsidered first.

The order that the coefficients during each bit-plane are processed arefrom the low resolution to the high resolution (from low frequency tothe high frequency). The coefficient subband coder within each bit-planeis from the high level (low resolution, low frequency) to the low level(high resolution, high frequency). Within each frequency subband, thecoding is in a defined order. In one embodiment, the order may be rasterorder, 2×2 block order, serpentine order, Peano scan order, etc.

In the case of a four level decomposition using the codestream of oneembodiment of the present invention, the order is as follows:

-   -   4-SS, 4-DS, 4-SD, 4-DD, 3-DS, 3-SD, 3-DD, 2-DS, 2-SD, 2-DD,        1-DS, 1-SD, 1-DD

One embodiment of the context model used in the present invention isdescribed below. This model uses bits within a coding unit based on thespatial and spectral dependencies of the coefficients. The availablebinary values of the neighboring coefficients and parent coefficientscan be used to create contexts. The contexts, however, are causal fordecodability and in small numbers for efficient adaptation.

The present invention provides a context model to model the bitstreamcreated by the coefficients in the embedded bit-significance order forthe binary entropy coder.

FIG. 13 shows the neighborhood coefficients for every coefficient of acoding unit. Referring to FIG. 13, the neighborhood coefficients aredenoted with the obvious geographical notations (e.g., N=north,NE=northeast, etc.). Given a coefficient, such as P in FIG. 13, and acurrent bit-plane, the context model can use any information from all ofthe coding unit prior to the given bit-plane. The parent coefficient ofthe present coefficient is also used for this context model.

The head bits are the most compressible data. Therefore, a large amountof context, or conditioning, is used to enhance compression.

Context Model—Transform Style

One embodiment of the context model of the present invention isdescribed below. This model uses bits within a coding unit based on thespatial and spectral dependencies of the coefficients. The availablebinary values of the neighboring coefficients and parent coefficientsmay be used to create contexts. The contexts, however, are causal fordecodability and in small numbers for efficient adaptation.

In the present invention, the sign bit context model comprises encodingthe sign after the last head bit. There are multiple contexts for thesign depending on whether the N coefficient is positive, negative or thesign is not yet coded.

Child-Based Order

In one embodiment, the bit-planes of the individual frequency bands arecoded in the order SS DS_(n) SD_(n) DD_(n) . . . DS_(i) SD_(i) DD_(i) .. . DS₁ SD₁ DD₁. In one embodiment, the order within a frequency bandfor the look-ahead, as well as the individual coding, is the child-basedorder. Child-based order is a scan order through the two-dimensionalimage, similar to raster order except for the two line, two-by-two blockorder. Consider scanning a “parent” frequency band in raster order. Eachcoefficient will have four children. These children are orderedtop-left, top-right, bottom-left, then bottom-right. Then the orderreturns to the left side and starts the next two lines finally ending inthe lower right corner. No lines are skipped. If there are an odd numberof lines, the last line is in simple raster order. FIG. 14 shows thisordering.

Frequency Band Bit-Plane Look-Ahead

In one embodiment, the bits in the importance level are coded in thefrequency band order. When coding a bit-plane of a frequency band, thefirst output bit indicates whether or not the entire bit-plane is zero.If it is 0, then a zero bit is delivered to the entropy coder. There isa single context for frequency band bit-plane look-ahead. The coderproceeds to the next frequency band bit-plane.

If there is at least one 1 bit, then a one bit is delivered to theentropy coder and the bit-plane is coded as described below. (Note thata one bit can be delivered to the entropy coder even if the bit-plane isall zeros. In this case, the bit-plane is coded as described below.)This pass is required for every bit-plane which could contain data.There is no bit coded for frequency bands which, because of alignmentand coefficient depth, cannot have a one bit at the current importancelevel, or for frequency bands which contain no coefficients.

In one embodiment, neighboring and parent DS, SD and DD coefficientsthat do not exist due to edges of tiles are treated as 0. This treatmentalso applies to the process of determining whether to attemptlook-ahead, the post look-ahead head bit context model and the head bitcontext model.

Many of the context models described herein make use of bits from othercoefficients (neighboring coefficients in the same frequency band, andthe parent coefficient, etc.). In one embodiment, the bits that areexamined depend on the type of neighbor. If the most significant bit ofa coefficient is being coded, then the reference bit in the parent isthe second most significant bit, the reference bit in the west,north-west, north, and northeast neighbors is also the most significantbit, the reference bit in the east and south neighbors is the bit moresignificant than the most significant bit, and thus is assumed to bezero. The reference bit to use is shown in FIG. 15. When coding theleast significant bit the parent is assumed to have another bit belowthe actual least significant bit which is zero. If the reference bit ofthe parent coefficient is actually in a lower importance level than thecurrent bit, the parent is assumed to be a zero head bit (the parent isnot used).

16-bit Look-Ahead

In the present invention, a look-ahead is used. This look-ahead isdesigned to reduce the redundancy of decisions coded by the coder. Thedetermination of whether the look ahead is used is based solely oncasual and deterministic data. If not, no data is coded and eachcoefficient is coded independently as described in the followingsections. If the look-ahead is attempted and is successful, a zero bitis coded with a look-ahead context and the 16 coefficients are skipped.Otherwise, a one bit is coded with a look-ahead context and eachcoefficient is coded as described in the following description. (Notethat a one bit can be coded even if the look-ahead was successful. Thisallows the encoder to bypass the look-ahead process.)

In one embodiment, the decision to attempt the look-ahead depends onwhether any one bits have been encountered in the 16 target coefficients(in child-based order), in the 4 parents of those coefficients, or inthe 8 northern neighbors of those coefficients. The look-ahead isattempted if the reference bits in the parents, the reference bits inthe northern neighbors, and bits in the previous bit-plane of the 16target coefficients are all zero head bits. To clarify, not only are allthe reference bits zero, but all bits more significant than thereference bits are zero. FIG. 16 shows these coefficients. At the edgeof the tile or image, there may not be 16 target coefficients availablein two rows; in this circumstance, no look-ahead is attempted. (QuestionD2 of the flow chart in FIG. 20 is answered no).

Note that if the parents are unavailable (due to alignment ornonexistence) or are not used then it is assumed that no one bits areever encountered. In one embodiment, this consideration is also appliedto the highest level DS, SD, and DD coefficients, because the SScoefficients are not used as parents. For these frequency bands, theparent is considered unavailable. Further, if there are no northernneighbors (e.g., the first line of the tile), then it is assumed thatthese unavailable bits are zero.

If the decision is to proceed with the look-ahead, the 16 bits of thecurrent bit-plane of the target coefficients are examined to see if theyare all zero head bits. If so, then a zero bit is coded with a contextthat consists of the last look-ahead attempted in the current frequencyband at the current importance level. If this is the first look-aheadattempted in the current frequency band at the current importance level,then it is assumed that previous look-ahead was successful (assumes azero was coded previously). If the 16 bits are not all zero head bits, aone bit is coded with the same context. It should be noted that othernumbers of coefficients other than 16 may be used, such as 8 or 32.Other selections may be based on available memory or may be based on thelocation of reference bits that are not zero head bits.

Post Look-Ahead

If the look ahead is attempted and fails, or is not attempted, the 16(or fewer) bits are coded individually. Each coefficient has head bitsdecoded until the first one bit occurs, then the sign bit is codedimmediately thereafter. After that, the coefficient is in the tail.

The coding is similar for the two cases: 1) look-ahead failed, 2)look-ahead not attempted. However, in one embodiment, different contextsare used, and in the first case, it is known that all coefficients to bedecoded are head bits.

Post Look-Ahead Head Bit Context Model

If the look-ahead is attempted and fails, then a few facts are known.First, the parent and northern neighbor coefficients of the top 8 areall in the zero head state. Second, so are all the target coefficients.Finally, there is at least 1 one bit among the target bits. Because thisinformation is so different from not attempting the look-ahead,different contexts are used for coding these head bits. Thus, adifferent context model is used for these bits so as not to combine verydifferent statistics.

To form the context model, certain values are derived from theneighboring pixels. It is clear that only some information from theparent and nearest neighbor coefficients can be used because, otherwise,the context model would be prohibitively large. Table 4 shows the typesof information used from each coefficient. FIG. 17 shows thecontributions of the neighborhood coefficients. TABLE 4 Coefficientcontext information Definition Type A Type B Reference bit is a 0 headbit 00 0 Reference bit is the head bit with a value 1 01 1 Reference bitis the first or second tail bit 10 1 Reference bit is a tail bit afterthe second tail bit 11 1

The present invention takes special steps at tile boundaries. In oneembodiment, if there are fewer than 16 coefficients (8 columns), then nolook-ahead is attempted. Also, if there is only one line at the bottomof a tile, then no look-ahead is attempted.

Head Bit Context Model When Look-Ahead Not Attempted

The bits in the importance level are coded in the frequency band orderdefined. Within each frequency band, the bits are coded in child-basedorder. The head bit context model portion of the transform-style contextmodel depends on two sources of information:

-   -   the parent coefficient, if signaled, 2 bits,    -   the nearest neighboring coefficients, 7 bits,

These 9 bits form a unique number that signals a particular state calleda context. This context is delivered to the FSM coder and is used toadapt the probability estimate for coding. Clearly, the information usedfor this context is casual; that is it is available at the time ofdecoding. It is also clear that only some information from the parentand nearest neighbor coefficients can be used. Table 4 shows the typesof information used from each coefficient.

When used the parent coefficient contributes 2 bits to the context (typeA information in Table 4). The parent coefficient is defined as thecoefficient one transform level up (in the same frequency band) thatcovers the same spatial area. Coefficients in the highest level of eachfrequency band do not have a parent (SS coefficients are not used asparents), and the contribution is defined to be 0. FIG. 18 shows anexample of a parent.

In the case where lower transform level coefficients are aligned suchthat the reference bit in the parent is below the current bit, theparent is not used for the context (the 2 bit contribution to thecontext is always zero). Also, the parent can be signaled as unused,which is useful for truly random access into the frequency bands of acoefficient. Also, coefficients that do not have parents do not useparents.

The contributions to the context from the neighbor coefficients areshown in FIG. 19. The types refer to Table 4.

Sign Bit Context model

The sign bit of every coefficient is coded immediately after the lasthead bit (the first one bit) of that coefficient. The sign bit is partof the importance level that contains the last head bit. In oneembodiment, the context for the sign bit is different from the head bitcontext and the tail bit context, consisting of three states based onthe current sign bit value of the north neighboring coefficient. Ifthere is no northern neighbor, the sign is unknown. The last head bitand the sign bit should be considered an atomic unit. Table 5 shows thecontext used for the sign bits. The same sign bit contexts are usedwhether the sign bit is being coded after a head bit or a postlook-ahead head bit. TABLE 5 Sign bit context information DefinitionBits Sign of north coefficient is unknown 0 Sign of north coefficient ispositive 1 Sign of north coefficient is negative 2Tail Bit Context Model

The tail bit context model is different from the head or sign bitcontext models. It consists of three states based on how many tail bitsthere have been in the current coefficient. Table 6 shows these values.TABLE 6 Tail bit context information Definition Bits Reference bit is1st tail bit 0 Reference bit is second or third tail bit 1 Reference bitis more than third tail bit 2Steps For Modeling Each Frequency Band Bit-Plane

One embodiment of the process for modeling each bit-plane of eachfrequency band of each importance level can be viewed graphically inFIG. 20. The decisions made are in Table 7 and the bits and context sentto the coder are in Table 8. In one embodiment, there are a total of 5independent contexts. TABLE 7 Decisions in the context model flow chartDecision Question D1 Are all the target bits in the frequency band zero?D2 Are there 16 coefficients left in the current two rows and if so, arethe 4 parents, and 8 northern neighbors zero head bits in the currentbit-plane, and the 16 target coefficient bits zero head bits in theprevious bit-plane? D3 Are the 16 target bits zero head bits in thecurrent bit-plane? D4 Was the head bit just coded a one bit? D5 Has theend of the 16 bits or the lines been reached? D6 Has the end of thefrequency band been reached? D7 Is the target bit a head bit? D8 Was thehead bit just coded a one bit? D9 Have the end of the 16 bits or thelines been reached?

TABLE 8 Coding in the context model flow chart Code Bit to code ContextC1 All bits==0?0:1 Frequency band (1 context) C2 All bits==0?0:1Look-ahead (2 contexts) previous look-ahead result previous C2 bit coded(1 bit) C3 target bit Post look-ahead bit (16 contexts) neighborinformation (4 bits) C4 sign bit Sign bit (3 contexts) northern neighborsign bit (1 bit) or unknown C5 target bit Head bit (512 contexts) parentinto (2 bits) neighbor information (7 bits) C6 target bits Tail bit (3contexts) target coefficient information (depth into tail) (2 bits)

An alternative embodiment of a context model, including an embodiment ofa sign/magnitude unit that converts input coefficients into asign/magnitude format, is described in U.S. patent application Ser. No.08/498,695, filed Jun. 30, 1995and entitled “Method and Apparatus ForCompression Using Reversible Wavelet Transforms and an EmbeddedCodestream” and U.S. patent application Ser. No. 08/498,036, filed Jun.30, 1995 and entitled “Reversible Wavelet Transform and EmbeddedCodestream Manipulation” and also U.S. patent application Ser. No.08/642,518, filed May 3, 1996 and entitled “Compression andDecompression with Wavelet Style and Binary Style Including Quantizationby Device-Dependent Parser” and U.S. patent application Ser. No.08/643,268, filed May 3, 1996 and entitled “Compression/DecompressionUsing Reversible Embedded Wavelets”.

The context model provides contexts for entropy coding of the data. Inone embodiment, all the entropy coding performed by the presentinvention is performed by binary entropy coders. A single coder may beused to produce a single output code stream. Alternately, multiple(physical or virtual) coders may be employed to produce multiple(physical or virtual) data streams.

Binary-Style Context Model

The modeling for the binary-style is similar to the transform-stylecontext model. Among the differences, however, is that the binary-stylecoefficients are unsigned numbers: there is no sign bit or distinctionbetween head and tail. FIG. 36 shows the flow of this context model.

Frequency Band Bit-Plane Look-Ahead

The bits in the importance level are coded in the frequency band orderdefined above. When coding a bit-plane of a frequency band (possiblypart of an importance level) is coded, the first output bit of thecontext model determines whether or not the entire bit-plane is thesame. If all bits are the same, then a 1 bit is delivered to the entropycoder; otherwise, a 0 bit is coded. Then, one bit is delivered to theentropy coder, indicating what that one bit is: 0 or 1. There is aseparate context for frequency band bit-plane look-ahead, and a contextfor the value of the bit. The coder proceeds to the next frequency bandbit-plane.

If there are two different bits, then a 1 bit is delivered to theentropy coder and the bit-plane is coded as described below. Note that a1 can be is delivered to the entropy coder even if the bit-plane isconstant. In this case, the bit-plane is coded as described below. Thisbit signaling the look-ahead is required for every frequency bandbit-plane.

16-bit Look-Ahead

This 16 bit look-ahead checks whether the next 16 bits (in the sameline) are all the same; if so, a 0 bit is delivered to the entropycoder. Then a 0 or 1 is delivered to indicate which bit the bits werethe same as. There is a separate context for N bit look-ahead. If, atthe end of a frequency band, fewer than 16 bits remain, those remainingbits are treated in this same manner. If all of these conditions are nottrue, a 1 bit is delivered to the entropy coder.

Spatial Context Model

The bits in the importance level are coded in the frequency band orderdefined above. Within each frequency band, the bits are coded in rasterorder. The context model depends on 7 neighboring pixels within the samefrequency band.

One bit from each of these pixels forms a unique number that signals aparticular state called a context. This context is delivered to the FSMcoder and is used to adapt the probability estimate for coding. Theinformation used for this context is causal; it is available at the timeof decoding.

The diamonds from FIG. 36 are described in Table 9. TABLE 9 Decisions inthe context model flow chart Decision Question Description D1 Are allthe target bits (bits being coded now) in the frequency band are thesame? D2 Are the 16 target bits in the current bit-plane the same? D3Have the end of the 16 bits or the lines been reached? D4 Has the end ofthe frequency band been reached?

The coding boxes from FIG. 36 are described in Table 10. TABLE 10 Codingin the context model flow chart Code Bit to code Context Description C1All Frequency band (1 context) bits==x?0:1 C2 x Equivalent Bit (1context) C3 All Look-ahead (2 contexts) bits==y?0:1 C4 y Equivalent Bit(if its the same, is it a 1 or 0) (1 context) C5 target bit Bit (128contexts) 7 neighboring bitsNeighbor Coefficients

The neighbor coefficients that contribute to the context are shown inFIG. 37. Each contribution the bit value at that coefficient at thecurrent bit-plane. Note that because each frequency band is asub-sampling of the original images, the pixels used in the templatewill not be immediately adjacent in the original image.

The Encoding and Decoding Process of the Present Invention

FIG. 21 illustrates one embodiment of the encoding process of thepresent invention. Referring to FIG. 21, the encoding process beginswith processing logic acquiring input data for a tile (processing block2101).

The processing logic then determines whether binary coding needs to beperformed (processing block 2102). If binary coding is to be performed,the process continues to the processing block 2111 where the processinglogic performs Gray coding on the input data, and models each bit ofeach coefficient with a binary style context model (processing block2112). The processing continues to processing block 2108.

If binary coding is not to be performed, the process continues toprocessing block 2103 where the processing logic applies a reversiblefilter to the data. After applying the reversible filter, the processinglogic tests whether there is another pyramid level desired (processingblock 2104). If another pyramid level is desired, the processing logicapplies the reversible filter to the LL coefficients (processing block2105) and the processing moves back to a processing block 2104 where thetest is repeated. If another level of decomposition is not desired, theprocess continues to processing block 2106 where the processing logicconverts the coefficients to sign-magnitude form. Thereafter, theprocessing logic models each bit of each coefficient with the horizoncontext model (processing block 2107), and the process continues toprocessing block 2108.

At processing block 2108, the processing logic codes each bit of eachcoefficient. The processing logic then transmits and stores each decodeddata (processing block 2109).

The processing logic then tests whether more tiles are used in the image(processing block 2110). If more tiles are in the image, the processinglogic looks back to processing block 2101 and the process is repeated;otherwise, the process ends.

FIG. 22 illustrates one embodiment of the decoding process of thepresent invention. Referring to FIG. 22, the process begins by acquiringcoded data for a tile (processing block 2201). Next, the processinglogic entropy decodes the decoded data (processing block 2202). Theprocessing logic then tests whether the data is to undergo binarydecoding (processing block 2203). If the data is to undergo binarydecoding each bits, the process continues to processing block 2211 wherethe processing logic models each bit of each coefficient with a binarystyle context model and performs inverse Gray coding on the data(processing block 2212). After the inverse Gray coding, the processcontinues to processing block 2209.

If binary decoding is not to be performed, and the process continues toprocessing block 2204 where the processing logic models each bit of eachcoefficient with the context model. Then, the processing logic convertseach coefficient to the proper form for filtering (processing block2205) and applies a reversible filter to the coefficient (processingblock 2206).

After applying the reversible filter, the processing logic tests whetherthere is another pyramid level (processing block 2207). If there isanother level of decomposition, the process continues to processingblock 2208 where the processing logic applies a reversible filter to thecoefficient and the process loops back at the processing block 2207. Ifanother level of decomposition is not required, then the processcontinues to processing block 2209 where the reconstructed data iseither transmitted or stored.

Next, the processing logic tests whether there are more tiles in theimage (processing block 2210). If there are more tiles in the image, theprocessing loops back to processing block 2201 and then the process isrepeated; otherwise the process ends.

Entropy Coding

In one embodiment, all the entropy coding performed by the presentinvention is performed by binary entropy coders. In one embodiment,entropy coder 104 comprises either a Q-coder, a QM-coder, a finite statemachine coder, or a high speed parallel coder, etc. A single coder maybe used to produce a single output code stream. Alternately, multiple(physical or virtual) coders may be employed to produce multiple(physical or virtual) data streams.

In one embodiment, the binary entropy coder of the present inventioncomprises a Q-coder. For more information on the Q-coder, seePennebaker, W. B., et al., “An Overview of the Basic Principles of theQ-coder Adaptive Binary Arithmetic,” IBM Journal of Research andDevelopment, Vol. 32, pg. 717-26, 1988. In an alternate embodiment, abinary entropy coder uses a QM-coder, which is a well known andefficient binary entropy coder. It is particularly efficient on bitswith very high probability skew. The QM-coder is used in both the JPEGand JBIG standards.

The binary entropy coder may comprise a finite state machine (FSM)coder. Such a coder provides the simple conversion from a probabilityand an outcome to a compressed bit stream. In one embodiment, a finitestate machine coder is implemented using table look-ups for both decoderand encoder. A variety of probability estimation methods may be usedwith such a finite state machine coder. Compression is excellent forprobabilities close to 0.5. Compression for highly skewed probabilitiesdepends on the size of the lookup table used. Like the QM-coder, it isuseful with embedded bit streams because the decisions are coded in theorder of occurrence. There is no possibility for “carry-over” problemsbecause the outputs are defined by a lookup table. In fact, there is amaximum delay between encoding and the production of a compressed outputbit, unlike the Q and QM coders. In one embodiment, the finite statemachine coder of the present invention comprises a B-coder described inU.S. Pat. No. 5,272,478, entitled “Method and Apparatus for EntropyCoding”, issued Dec. 21, 1993. In another embodiment, the finite statemachine coder comprises a coder described in U.S. patent applicationSer. No. 08/719,819, entitled “Apparatus and Method for Performing M-ARYFinite State Machine Entropy Encoding,” filed Sep. 26, 1996.

In one embodiment, the binary entropy coder of the present inventioncomprises a high speed parallel coder. Both the QM-coder and the FSMcoder require that one bit be encoded or decoded at a time. Thehigh-speed parallel coder handles several bits in parallel. In oneembodiment, the high speed parallel coder is implemented in VLSIhardware or multi-processor computers without sacrificing compressionperformance. One embodiment of a high speed parallel coder that may beused in the present invention is described in U.S. Pat. No. 5,381,145,entitled “Method and Apparatus for Parallel Decoding and Encoding ofData”, issued Jan. 10, 1995.

Most efficient binary entropy coders are limited in speed by fundamentalfeedback loops. A possible solution is to divide the incoming datastream into multiple streams and feed these to parallel encoders. Theoutput of the encoders are multiple streams of variable-length codeddata. One problem with this type of approach is how to transmit the dataon a single channel. The high speed parallel coder described in U.S.Pat. No. 5,381,145 solves this problem with a method of interleavingthese coded data streams.

Many of the contexts used in the present invention are fixedprobability, which makes a finite state machine coder, such as theB-coder especially useful. Note when a system using probabilities closeto 0.5, both high speed parallel coder disclosed above and the finitestate machine coder operate with more efficiency than the Q-coder. Thus,both have a potential compression advantage with the context model ofthe present invention.

In another embodiment, both a binary entropy coder and a fast m-arycoder are used. The fast m-ary coder may be a Huffman coder.

Lossy Compression Reconstruction

Lossy Coefficient Reconstruction

After coefficients have been quantized, there is a range of numberswithin the legitimate reconstruction values. In such a case, the lowerorder (or bottom) bits of a coefficient are typically unknown and bitvalues for these lower order bits must be assigned. In one embodiment,the present invention performs lossy reconstruction of the quantizedvalues by truncating values to a predetermined set of integer values.For instance, in one embodiment, all coefficients between 0 and 31 arequantized to 0, all coefficients between 32 and 63 are quantized to 32,and so on. Thus, in this case, all of the unknown bits of thecoefficients are replaced with all zeros. FIG. 23 illustrates a typicaldistributions of coefficients without quantization.

In another embodiment, a value in the middle of each region may providea more accurate value to represent the group of coefficients. Forinstance, all coefficients between 64 and 127 are quantized to 96 (oralternatively, 95). The point to which the values are quantized isreferred to as the reconstruction point.

In still another embodiment, the value 0.375 (⅜) from the lower bound ofeach region may be used. For instance, all coefficients between 64 and127 have a reconstruction point of 88. Any value may be selected basedon the specific image(s), distribution of data, desired result, or othercriteria.

Due to the difference between images, the resulting distributions mighthave skewed shapes. For instance, compare curves 2701 and 2702 in FIG.23.

In the present invention, the reconstruction point is selected based onthe distribution. In one embodiment, the distribution is estimated and,based on that estimate, a reconstruction point is chosen. The estimateis generated based on the data that is already known. In one embodiment,a histogram of quantized coefficients may be used to make a prediction.

The variance in the distribution may be determined during encoding. Byproviding this variance to the decoder during decoding, a betterprediction may be made for selecting a reconstruction value. A singlevariance may be used for all quantized coefficients in a frequency band.In one embodiment, the variance is signaled to the decoder. Suchsignaling may be by a separate signal or by providing the variance in atag, such as a comment tag.

Note that the selection of the reconstruction point could inject noiseinto the non-zero coefficients. Depending on what reconstruction pointis selected, different amounts of energy may be injected into the image.In one embodiment, different reconstruction points are used fordifferent pyramid levels or different subbands.

In one embodiment, prior to gathering data, a default reconstructionpoint may be used. Thus, the present invention provides an adaptivemethod of performing lossy reconstruction. Further, the presentinvention is a non-iterated method of improving the coefficientreconstruction.

To compensate for the non-uniform usage of the numeric range due todifferent distributions, the present invention provides for$\begin{matrix}{s^{2} = {{sample}\quad{variance}}} \\{Q = {Quantization}} \\{\sigma^{2} = {\frac{2}{\alpha^{2}} = {{True}\quad{variance}}}} \\{\alpha = {{- \frac{1}{Q}}{\ln\left\lbrack \frac{Q^{2} + {2s^{2}} - \sqrt{Q^{4} + {8Q^{2}s^{2}}}}{2\left( {s^{2} - Q^{2}} \right)} \right\rbrack}}}\end{matrix}$where S² is the sample variance measured by the decoder based on thedata available and Q is the quantization which is known to the decoder.Then correct non-zero coefficients by moving them away from 0$\begin{matrix}\left. {i\quad Q}\rightarrow{{i\quad Q} + \left\lbrack {\frac{1}{\alpha} - \frac{Q}{{\mathbb{e}}^{\alpha\quad Q} - 1}} \right\rbrack} \right. & {i > 0} \\\left. {i\quad Q}\rightarrow{{{+ i}\quad Q} - \left\lbrack {\frac{1}{\alpha} - \frac{Q}{{\mathbb{e}}^{\alpha\quad Q} - 1}} \right\rbrack} \right. & {i < 0}\end{matrix}$where i equals any integer.

In one embodiment, after all decoding is completed, every non-zerocoefficient is adjusted to a reconstruction level. This requiresreading, and perhaps modifying and writing each coefficient.

In another embodiment, as each bitplane of each coefficient isprocessed, if the coefficient is non-zero, the proper reconstructionvalue of the coefficient is stored. When decoding stops, allcoefficients are already set to their proper reconstruction value. Thiseliminates the need for a separate pass though the memory for settingreconstruction levels.

Noise Injection

The present invention provides for injecting noise into data beingdecoded. In one embodiment, the data being decoded is image data thathas been quantized. In one embodiment, the quantized image datacomprises quantized coefficients. Quantization of wavelet coefficientsis essentially a low pass operation. For instance, the data may bequantized when only a portion of the data is decoded. Performing lowpass filtering on image data is acceptable unless texture is destroyed.The feel of this texture may be recaptured by injecting noise. Thus, thepresent invention injects noise into the an image as a function of thequantization.

In one embodiment, noise is injected into the image using only the zeroquantized coefficients. A random value may be added to the zeroquantized coefficients. The zero quantized coefficient may berepresented as a series of zero bits followed by a certain number ofunknown bits. These unknown bits are reconstructed by the addition ofrandom values. If these are four bits of unknown data, they may bereplaced with a number from 0 to 15. The higher the number, the greaterthe noise. The unknown bits are magnitude bits. The sign bit may also berandomly chosen, resulting in coefficients between −15 and 15.

Note that the added noise in level 1 transform DD coefficients onlyaffects four pixel values due to the reversible wavelet transform of thepresent invention. Therefore, the result of injecting noise does notcause noticeable artifacts with the neighboring pixels.

In one embodiment, noise is not injected into each of the zero quantizedcoefficients. For example, noise may only be injected into the pyramidlevel 1 DD coefficients. In another embodiment, noise is only injectedinto pyramid level 1 DS and SD coefficients.

In an alternate embodiment, the noise is a function of the noise in theoriginal image. In order to inject noise as a function of the noise inthe original image, the noise in the original image is quantified andprovided to the decoder. In one embodiment, a spatial map is made thatillustrates the energy distribution. In one embodiment, thedistribution, amplitude and deviation of the noise in the original imageare signaled to the decoder. This information may be signaled to thedecoder using a tag in the codestream or using separate signals or aseparate information path.

In an alternative embodiment, an alpha plane may be made showing wherethe noise should be placed. The alpha plane could be used in a mannersimilar to that of a blending plane where the plane would indicatedifferent locations in the image where different amounts of noise are tobe injected. That is, at one location (e.g., region) the plane mayspecify noise of a first type to be injected, while another location inthe alpha plane could indicate that a different amount of noise is to beinjected.

The noise that is added may be based on a distribution of values withinthe region surrounding 0. If the distribution of values is offset andnot centered around the 0 value, a bias may have to be added (orsubtracted) as well as the noise.

Detiling and Deringing

The Two, Ten-Transform allows for some advantages in reconstructionafter lossy compression. To reiterate, the Two, Ten-Transform is definedas follows:S _(n)=└(X _(2n) +X _(2n+1))/2┘B _(n) =X _(2n) −X _(2n+1)P _(n)=(3S_(n−2)−22S_(n−1)+22S_(n+1)−3S_(n+2)+32)/64D _(n) =B _(n) +P _(n)X _(2n) =S _(n)+└(D _(n) −P _(n)+1)/2┘ or S _(n)+└(B _(n)+1)/2┘X _(2n+1) =S _(n)−└(D _(n) −P _(n))/2┘ or S _(n) −└B _(n)/2┘

In one embodiment, for lossy compression, D is quantized. For creatingpreferred reconstructions, in some cases, D values are computeddirectly. In other cases, a preferred B value is determined, which isthen converted to a D value using P if necessary.

New Order for Two-Pass Inverse Transform

FIG. 24 illustrates a method for computing the inverse TT-transform thatis useful when using adaptive wavelet coefficient reconstruction. Theoutput of each of two passes are S coefficients. In one embodiment, theS coefficients are samples of images (pixel components). In analternative embodiment, the S coefficients are averages of multiplesamples of the image (“super pixel components”). The S components can betreated as image data in the spatial domain, allowing preferredreconstructions to be generated.

Preferred reconstructions are based on an image model. An image modelmight provide for sharp edges and smooth regions where there are notedges. An image model might provide texture information. An image modelmight be image independent or described with the compressed for aspecific image. Image models will be described below. The use of thetwo-pass inverse transform of the present invention reduces the need tosave intermediate results to compute the transform and determinequantization limits. In other words, the SS and SD coefficients consumedby the first pass do not have to be saved.

Referring to FIG. 24, the two-pass inverse transform process of thepresent invention begins with the first pass, pass 1, performing theinverse vertical transform on the SS and SD coefficients only. Thequantization of each SD coefficient controls the reconstruction limitsof two samples. The second pass, pass 2, operates on two lines ofcoefficients at a time and performs the inverse vertical transform on DSand DD coefficients and the inverse horizontal transform for twovertically adjacent S and D coefficient pairs. The second pass continuesuntil all the lines of the data undergo the two inverse transformoperations of the second pass. Note that the quanization of the DS andDD coefficients controls the reconstruction limits of four samples.Also, for the second pass, the DS coefficients from two above lines arefor computing the vertical inverse transform on DS and DD coefficients.

Reconstruction and Clipping

One embodiment of a procedure for creating a preferred reconstruction isas follows.

-   -   Analyze the coefficients and/or normal reconstruction (Optional)    -   FOR transform level=max_level DOWNTO 1        -   FOR each subband DO            -   FOR each SD coefficient DO                -   Compute prefered reconstruction                -   Clip to be consistent with quantization            -   Do pass 1 of inverse transform            -   FOR each DS and DD coefficient pair DO                -   Compute prefered reconstruction                -   Clip to be consistent with quantization            -   Do pass 2 of inverse transform

The first step of analyzing the coefficients is used when the preferredreconstruction is estimated from the coefficients. An example of this isdiscussed in conjunction with edge extraction below.

Clipping is an operation in which a particular value is set to a valuein a range (or one of the end points of the range) when the value isoutside the range. Clipping is necessary to assure that thereconstructed image results in the same quantized coefficients asspecified in the coded data. For a given quantized coefficient value Dwhich was Q bits that are unknown to quantization, the present inventiondetermines the minimum and maximum possible values of D which are usedto clip preferred reconstructions, when necessary. One embodiment of thefollowing code may be used to calculate minimum and maximum values.inline void twoten: :q_to_minmax (int d, int q) {  int n = (1<<q) −1; if(d ==0) {  min = −n;  max = n; } else if (d>0) {  min = d & (−n);  max =d | n; } else {  max = −((−d) & (−n));  min = −((−d) | n);  } }

In the exemplary code above, the “&” refers to logical ANDing operationand the “|” refers to a logical ORing. For clipping D values, theclipping routine, “clip” described below may be used. The “clip” routineis useful for SD and DS. In one embodiment, the “clip” routine describedmay also be used for DD coefficients; however, in an alternativeembodiment, a routine such as “clip_loose” may be better to allow someoff-by-one errors to compensate for the independent clipping of relatedDS and DD values. Note that the clip-loose routine calls the “clip”routine. The “flat” parameter is the value of D that results in bothsamples being reconstructed as identical, i.e., it is the “P” portion ofthe TT-transform. Due to different integer rounding, off-by-one errorsthat result in both samples reconstructed as identical are permitted.inline int twoten:: clip (int n, int min, int max) {  if (n < min)  return min;  if (n> max)   return max;  return n; } inline inttwoten:: clip_loose (int n, int min, int max, int flat) {  if (min−1 ==flat)   return clip (n, flat, max);  else if (max+1 == flat);   returnclip (n, min, flat);  else   return clip (n, min, max); }

For DS and DD values, preferred reconstruction values are usuallydetermined as a pair of “d” values. The “do_clip_pair” routine belowclips two D values, “a” and “b”, resulting in “a_clip” and “b_clip”.Note that the routine calls the clip-loose routine. inline voidtwoten::do_clip_pair Itwoten *tt_0ds, towten *tt_1dd, int vert_p) {  inta;  int b;  int s;  int d;  int s_clip;  int d_clip;  int a_clip;  intb_clip;  a = tt_0ds->d( );  b = tt_1dd->d( );  s = s_calc(a,b);  d = a −b + vert_p;  s_clip = clip_loose (s, tt_0ds->get_min( ),tt_0ds->get_max( ),  vert_p);  d_clip = clip_loose (d, tt_1dd->get_min(), tt_1dd->get_max( ),  vert_p);  a_clip = inverse_calc0 (s_clip,d_clip, vert_p);  b_clip = inverse_calc1 (s_clip, d_clip, vert_p); }

An embodiment of the inverse_calc0 and inverse_calc1 routines are asfollows: inline int s_calc (int s0, int s1) (return (s0+s1) >> 1: };inline int twoten::inverse_calc0(int s, int d, int p) {  return s + ((d− p + 1) >>1); } inline int twoten::inverse_calc1 (int s, int d, int p){  return s − ((d − p) >> 1); }Note that one of these routines is for the even samples and one is forthe odd samples.Reconstruction for Tile Boundaries

The present invention creates preferred reconstructions to eliminatetile boundaries. The present invention creates preferred reconstructionsby using information from the neighboring tiles to generate areconstruction that would be consistent with performing the transformwithout tile boundaries.

In one embodiment, with preferred reconstruction, the forward transformis performed independently on each tile. During decoding, it is auser/application choice to decide to decode, reconstruct an inversetransform independently or not.

The TT-transform wavelet allows reconstruction to maintain theadvantages of an overlapped transform even when tiles of an image aretransformed independently. When the TT-transform is performedindependently on tiles of an image, the tile boundary artifacts can beeasily removed.

Tile boundaries artifacts can be readily eliminated from TT-transformcoefficients because of the following reasons. The TT-low pass filter isunaffected by boundaries when there is an even number of samples,leading to accurate S coefficients. D coefficients, which are affectedby boundaries, only have limited spatial effect. Note that smoothness isdefined to be having a zero high pass filter response when the filter isapplied crossing the boundary. Therefore, smoothing may be performedeasily in the transform domain and easily limited to the amount allowedby the quantization.

In one embodiment, the present invention eliminates tile artifacts byreconstructing before each application of the transform. Exemplary codeto eliminate tile artifacts is as follows: for (level=levels; level > 0;level−−)  save DS and DD coefficients effected by boundary  reconstructSD coefficients to be smooth across boundary  for each tile   verticalinverse transform on tile  reconstruct D coefficients to be smoothacross boundary  for each tile   horizontal inverse transform on tile

The P portion of the inverse TS-filter that is a function of Scoefficients and is given below:P=(3S⁻²−22S⁻¹+22S₁−3S₂+32)/64.

FIG. 25 shows the weights used to compute P_(f) across tile boundaries(full-frame). FIG. 26 shows the weights used to compute P_(t) on asingle tile boundary with mirroring. Tile boundary artifacts are causedby the difference between P_(t) and P_(f). By using D=−(P_(f)−P_(t)), asmooth result is obtained, which must be consistent with the actualquantized coefficients.

The SD coefficient may be made consistent with the quantization, sincethe quantization is known. First, the present invention determines theminimum and maximum allowed values from the quantized SD value, giventhat the number of bit planes quantized Q bits are unknown. As describedabove, the minimum and maximum values may be determined according to thefollowing code: N = (1 << Q) −1 if (DS == 0) (  MIN = −N;  MAX = N; }else if (DS > 0) {  MIN = DS & (−N);  MAX = DS | N; } else {  MAX =−((−DS) & (−N));  MIN = −((−DS) | N); }

The MIN and MAX values are used to clip the result obtained fromcomputing −(P_(f)−P_(t)). compute Pf and Pt SMOOTH = −(Pf−Pt) if (SMOOTH< MIN) SD = MIN; else (SMOOTH > MAX) SD = MAX else SD = SMOOTH

The quantization of DS and DD coefficients is propagated through thevertical transform. Therefore, handling the horizontal transform isslightly more complicated. In order to do so, a pair of lines, denoted“a” and “b” that share DS and DD coefficients are considered at a time.These DS and DD values were saved prior to the inverse verticaltransform; therefore, they are still available. The values after thetransform are DA and DB. The minimum and maximum values for DS and DD(MIN_DS, MIN_DD, MAX_DS, MAX_DD) are computed in the same manner as theDS minimum and maximum values.

-   -   compute Pfa and Pta for first line    -   computer Pfb and Ptb for second line        SMOOTHa=−(Pfa−Pta)        SMOOTHb=−(Pfb−Ptb)        S=(SMOOTHa=SMOOTHb)>>1;        P=DD−DA+DB;        D=(SMOOTHa−SMOOTHb)+P    -   clip S using MIN_DS and MAX_DS    -   clip D using MIN_DD and MAX_DD        DA=S+((D−P+1)>>1);        DB=S−((D−P)>>1);

For images containing no high frequency information (all coefficientsother than SS coefficients are zero), the reconstruction for any tiling(with an even number of samples for every pyramid level) is the same asfor full frame.

Because the reconstruction only effects a small number of totalcoefficients for reasonable tiles, neither the computation cost not thememory cost is very high. However, there are a number of simplificationsthat could be made to reduce these costs. FIG. 27 illustrates weightsfor calculating P_(f)−P_(t) approximately. Off-by-one errors due todifferent integer rounding prevent obtaining exactly the same result asfull frame on images with no high frequency, but should make nodifference in practical applications. Another potential simplificationis to approximate all S coefficients in other tiles with SS coefficientsto reduce memory usage.

Because the TT-filter is lossless, tile artifact removal may beperformed as post-processing. An image may be decompressed without tileartifact removal construction. The location of the tile boundaries andthe quantization are saved for later use. When an image without tileartifacts is desired, the image may be transformed, and then using theinformation about tile locations and quantization, it can reconstructedwithout tile artifacts.

In many systems, simplifying decoding is important. The amount ofinformation from neighboring tiles that is required for preferredreconstruction is small. That information could be stored in a commentwith the coded data for each tile, allowing each tile to be decodedindependently. Also, there is no requirement that every coefficient begiven a preferred reconstruction. In one embodiment, only coefficientsthat were in certain transform levels or those which are quantized tozero may be given preferred reconstructions.

Reconstruction of Step Edges

The present invention provides for reconstruction of step edges using apredetermined number of coefficients. In one embodiment, only Scoefficients are used and the predetermined number is 5. FIGS. 28A and28B illustrate examples sets of five S coefficients where reconstructionto linear (smooth) or step edges, respectively, is appropriate. Thesolid lines illustrate the step size of the preferred reconstruction.

One embodiment of the process for determining whether to perform linearreconstruction or step edge reconstruction is as follows. Note that inthe process, a “B” value, the difference between two samples, iscomputed, and the coefficient, D, is equal to the addition of B+P.

The present invention attempts to reconstruct a step edge where theimage is not flat. In one embodiment, if the minimum and maximum Svalues differ by less than 10, the present invention does notreconstruct a step edge. Note, in an alternative embodiment, a thresholdof 10 need not be used and a different threshold may be used.

In reconstructing a step edge, the present invention computes thereconstruction B using the left three S values, referred to as “LEFT”and the reconstruction B using the right three S values, referred to as“RIGHT”. If either the LEFT or RIGHT computation is zero, then thepresent invention reconstructs using B=zero and exits.

If both the LEFT and RIGHT computations have the same sign, the presentinvention uses the reconstruction with the smaller magnitude and exits.If also LEFT and RIGHT computations differ in sign, then presentinvention exits and a typical reconstruction is used.

In one embodiment, of the present invention, the calculation of thereconstruction B is performed by first determining the differences. Thevalues Δa and Δb represent differences between the outer and inner pairsof S values. For example, see FIGS. 28A and 28B.

The present invention then tests whether |Δa|>|Δb|. If so, the presentinvention changes Δa to Δa=Δb.

There are two options to handle the case when Δa and Δb have differentsigns. The first option is to set Δa=0. The first option makes this caseimply a hard edge. The second option is to set Δa=Δb. The second optionmakes this case imply smooth (normal) Two, Ten transform reconstruction.

Next, the present invention sets x=Δa/Δb. If the sign of Δa and Δbdiffer, the present invention sets B=2 Δb (1+x); otherwise, the presentinvention sets B=2 Δb(1−¾x). Finally, for LEFT, the present inventionreturns B, while for RIGHT, the present invention returns −B.

For line art or graphic images where, all or at least most, edges arestep edges, this reconstruction procedure of the present invention canbe used for all transform levels. For natural images, thisreconstruction can be used only for high frequency transform levels.

Matched Filter Reconstruction

The goal of reconstruction using matched filters is to not useinformation from the other side of edges. When an edge is found, thepresent invention uses symmetric extension on the other side of theedge. For example, one set of filter coefficients according to thepresent invention that can be used depending on whether or not an edgeis found are as follows. 3 −22 0 22 −3 no edge 0 −13 −9 25 −3 edge toleft 0 0 −48 64 −16 edge to left 3 −25 9 13 0 edge to right 16 −64 48 00 edge to right

The present invention may use the following matched filters to findedges in sets of five S coefficients. The order of these filterscorresponds to the filters above. 1 −4 4 −4 1 0 −1 3 −3 −1 0 0 2 −3 1 1−3 3 −1 0 1 −3 2 0 0In one embodiment, a filter coefficient set with extension is used if:

-   -   1) its corresponding matched filter has the minimum magnitude        response.    -   2) its corresponding in matched filter response magnitude is        less than a threshold. (for example, 48).    -   3) for a 4-tap matched filter, the corresponding 3-tap matched        filter response magnitude must also be less than the threshold.

The matched filter reconstruction of the present invention may beperformed on all frequency levels or just high frequency levels.

Edge Extraction

A goal of preferred reconstruction is to have sharp edges and toeliminate ringing near edges. In one embodiment, the present inventionidentifies where the edges in a reconstructed image are and then usesmultiscale information from different amounts of Gaussian smoothing.

Edge extraction uses a difference of gaussian technique as shown in FIG.29. To locate edges with single pixel accuracy, no subsampling isperformed. While any type of low pass filter could be used, separablehorizontal and vertical filters are convenient for computationalefficiency. Using odd length filters results in no phase shift. A goodchoice is the following filter taps (followed by division by 16):

-   -   1 4 6 4 1

In the present invention, the edge extraction process initiallysubtracts pairs of adjacent pixels (horizontally or verticallyadjacent). The present invention saves, as potential edge locations,differences with magnitude that are greater than at least one neighborin the direction of the difference. Other, smaller, differences areignored. Also, differences below a threshold (for example 8) areignored.

Referring to FIG. 29, a low pass filter comprises a low pass horizontalinverse transform filter and a low pass vertical transform filter. Theoutput of lowpass filter 2902 is used as the input of the low passfilter in the next resolution. Filter outputs are processed by thehorizontal difference 2903, used for local maximum 2905, and thevertical difference 2904, used for local maximum 2906. Edge locations indifferent resolutions are compared. Edge locations (outputs of the localmaximum units) are saved if in the next higher or lower resolution, thecorresponding location or one of its four neighbors is a saved potentialedge location with the same sign. Real edges happen in the same place indifferent scales and artifacts don't (i.e., the artifacts don't lineup).

One use of the location of edges is for adaptive filtering thatpreserves edges while reducing ringing artifacts. One way ofimplementing this is to use the 5-tap low pass filter described abovesection on five sample windows with no edge. If there is an edge in thecenter of the five sample window, the sample is used unchanged (nofiltering). When there is one or more edges in other positions in thewindow, only the samples and taps in the center of the window notincluding or outside of the edge(s) are used and the filter's divisor isthe sum of the taps used. Multiple (e,g., 2) iterations of the filtercan be used. This filter tends to sharpen edges. Pixels next to edgesare influenced by pixels farther from the edge, which tends to opposeprevious blurring of information from the other side of the edge.

Using a Preferred Spatial Domain Image

The present invention creates a preferred reconstruction as follows.First, the present invention creates a preferred spatial domain image.For example, the edge preserving adaptive filter method described abovemay be used. Next, the present invention performs a partial wavelettransform of the preferred image. The horizontal transform is performednormally. Only the S outputs of the horizontal transform are processedby the vertical transform. The D outputs of the horizontal transform arenot processed. See FIG. 30. Then, the present invention clipscoefficients from the partial transform to the valid range of thequantized coefficients to create preferred reconstruction.

The spatial domain preferred image may be updated in regions wherecoefficients are clipped and the process iterated if desired.

Low Pass Coefficients Covering Edges

The edge finding method of the present invention locates edges withsingle pixel accuracy. S or SS coefficients in the inverse wavelettransform correspond to multiple pixels. The number of pixels is2^(2*level-1) for S coefficients and is 2^(2*level) for SS coefficients.If any of the pixel locations corresponding to an S or SS coefficientcontain an edge, that coefficient is considered to be an edge.

For the S or SS coefficients in the center of a five sample window fordecomposition levels greater than 1, using the full(2^(level))(2^(level)) or (2^(level))(2^(level-1)) edge search regionmight not be desirable. Detecting an edge in the center of the windowwhen it is really on the boundary can reduce the opportunity for findinga good reconstruction. Instead a smaller search region in the directionof the transform such as (2^(level-1)*2 ^(level)) or(2^(level-1))(2^(level-1)) or (2^(level-2))(2^(level-1)) may be used.Using −1 for level 2 and −2 for levels greater than 2 may be used.

Once the S or SS coefficients that cover edges are known, mirroring canbe used for those coefficients on or on the other side of edges.

Clipping Reconstructions

Particularly for reconstructions that attempt to sharpen edges, clippingreconstructions so they do not exceed the bounds of neighboring Scoefficients is useful and reduces ringing. FIG. 31 illustrates aclipping reconstruction. In one embodiment, either the immediateneighbors of the center sample may be used or the closest in value tothe center sample of the two neighbors on either side.

FIG. 31 illustrates that sample value (e.g., pixel) range of thereconstructed value for the center S coefficient exceeds the value ofthe fourth S coefficient. In such a case, if the reconstructed value isgreater than the neighboring sample, such as S coefficient four, itsvalue is clipped to the value of the neighboring pixel, e.g., Scoefficient four. Similarly, the possible reconstructed value of thecenter sample does not include the value of the second S coefficient,which is next to the center pixel in FIG. 31. In such a case, theallowable value of the center sample may be extended to the value to bethe value of the that sample, so clipping is not indicated. Note,however, in either case, changing one side of the allowable range of thereconstructed value causes a corresponding change in the other side ofthe range. For instance, if the upper range of the reconstructed valuefor the sample must be clipped to be no greater than the fourth samplevalue, then likewise the range below the center sample is also reduced.Therefore, in this manner, the present invention provides for clippingreconstruction that do not exceed the bounds of the neighboringcoefficients and reduced ringing.

Texture Extraction

Wavelets are good at representing edges and smooth regions. Texture isdifficult to represent because it must be represented as many smalledges. There are many ways to represent texture, for instance, see WP.K. Pratt, Digital Image Processing, John Wiley and Sons, 1978. Thepresent invention provides a representation of texture which can bebetter than the texture model inherent in DCT based compression methodssuch as JPEG. Sinusoidal grids are used.

To enable texture extraction, the following operations occur duringencoding. The present invention generates a residue image containingtexture not well represented by wavelets. To do so, first, the presentinvention generates a reconstructed image after quantization. Forexample, the present invention quantizes one MSE importance level morethan the approximate target compression. Second, the present inventionsubtracts the quantized, reconstructed image from the original image.This is the residue image.

Next, the present invention models the texture in the residue image. Oneway is to find the 1D sinusoid with arbitrary rotation in 2D andarbitrary phase and arbitrary frequency that has the largest correlationwith each 16×16 block of the residue image. Then the present inventionoutputs the texture model parameters as a comment in the compressed datastream.

In the process above, the most important step is modeling the texture.Correlating with sinusoids is computationally expensive. Autocorrelationis used to limit the search space for sinusoids. The 2D autocorrelationis computed for blocks for the residue image. The half the period of asinusoid corresponds to negative autocorrelation (see FIG. 32). Only thefirst region of negative correlation around zero is of interest.

The region of negative autocorrelation can be searched as follows foreach integer x_(i),y_(i) position in the region. The step size (step)can be ¼ or ⅛. The threshold can be 4.75. Other step sizes andthresholds may be used.

FOR x=x_(i)−1+step TO x_(i) STEP step

-   -   FOR y=y_(i)−1+step TO y_(i) STEP step        -   r=sqrt (x²+y²)        -   IF r<threshold THEN            -   x=x/r            -   y=y/r            -   correlate with sinusoid r, x, y

The value of r is half the period of the sinusoid. The parameters x andy are a unit vector which specifies the angle to rotate the 1D sinusoid.These parameters are used both for determining the best match duringencoding and for generating texture during decoding.

FOR j IN vertical region to correlate with sinusoid

FOR i IN horizontal region to correlate with sinusiodθ=π*(i*x+j*y)/r

-   -   coef_c=cos θ*residue [i, j]    -   coef_s=sin θ*residue [i, j]    -   correlation=sqrt(coef_c²+coef_s²)

Using this model, the four parameters are stored as a comment in thecoded data stream such as x, y, coeff_c and coeff_s. The parameterscoeff_c and coeff_s can be quantized to 0.5 steps.

During decoding, the quantized image used to generate the residue iscreated. The information in the texture comment is used to generatesinusoids. The sinusoids are added to the quantized image, resulting ina preferred reconstruction image. The preferred reconstruction is usedthe preferred spatial domain size. If the wavelet coefficients availableto the decoder are more accurate than those used in the quantized imageused for generating the residue, some of the artifacts due to thetexture model are reduced.

Instead of using a block based texture model, a continuous or overlappedmodel may be used.

The model above is good for modeling stripes and similar 1D textures. Itcan be extended to handle 2D sinusoidal grids. In autocorrelation space,the area along the line perpendicular to x, y (and going through 0,0)can be searched for the first negative region. The length along thisperpendicular direction can be used to specify the half-period of thesinusoid in this direction. The region of negative autocorrelation mightbe used as a first step in other texture modeling methods. For example,it might be used to generate structured noise.

Special Buffer for a Tile

In one embodiment, the coding method of the present invention isimplemented in software running on one or more processors in a computersystem. In this type of system, the context model of the presentinvention examines many coefficients and uses many pointers to keeptrack of these coefficients that will be used to provide a context for acoefficient to be coded later temporally. The pointers point to memorylocations that contain the coefficients that are used for the context.Also, a set of offsets, both horizontal and vertical, are maintained todetermine how to update the pointers to point to the memory locationsfor the next context. The horizontal offset is the distance from thecurrent coefficient to the next coefficient to the east, and thevertical offset is the distance from the current coefficient to thesouth. These offsets are dependent on the memory and how thecoefficients are stored in the memory. The context model handles edgesusing special cases when the coefficients that are needed to provide acontext do not exist because of the presence of an edge or boundarycondition.

FIGS. 38A and 38B illustrate two examples of nine pointers that aremaintained to point to memory locations for determining the next fourcoefficients X, Y, U, and V. Referring to FIG. 38A, the nine pointersinclude pointers to coefficients N, X, U, and S. Farther to the left ofthe N, X and U coefficient are three additional coefficients NW, XW andUW. To the right are two additional coefficients XE and UE. The NX, Uand S pointers are used to access the locations shown in dashed boxes.FIG. 38B illustrates pixel values X another version in which the ninevalues are NW, XW and UW on the left side with NX, X and U forming amiddle column and NY, Y and V forming a column to the right. Note thatin this case, the S and E information is stored in signaling bits. Whenupdating the memory location of U, a determination is made as to what isoccurring with respect to S. When X is being coded, NX is updated, whicheventually becomes the U position.

To improve context modeling in a software driven computer systemimplementation, the present invention uses a special buffer for eachtile when providing contexts. The buffer is a block of memory that maybe a fixed or variable sized block of contiguous memory with the size ofthe block of memory being greater than the size of the tile (orfrequency band) stored or to be stored therein. For instance, if thetile is 256 coefficients in each dimension, the block of memory may be384 coefficients in each dimension. An example is shown in FIG. 33.Therefore, no matter what the tile size is, a larger area of memory isallocated to it. The block of memory does not have to be the same shapeas the tile stored within it. For example, a block of memory allocatedto a square tile does not have to be square itself.

The advantage of using a larger block of memory is that the pointers canbe replaced with a single pointer to a predetermined point in the tile(e.g., upper left corner) and a set of offsets. These offsets are fixedfor each subband. Thus, instead of 9 pointers and at least two offsetsin the case of the context model described above, only one pointer and aset of offsets are necessary. In one embodiment, one pointer to thenorthwest (NW) coefficient is used with two fixed offsets in thehorizontal and vertical for coefficient access. The correct number ofapplications of the offsets are used from the NW pointer to the other 9locations.

One benefit of such an approach is a reduction in the number ofregisters needed. The offsets need not be stored in registers or storagespace as part of the compiled code; they are constants.

Note that although the present invention replaces multiple pointers withone pointer (but less than that previously required) and multipleoffsets, it is apparent that more than one pointer could be used with alesser number of offsets depending on the implementation.

Another advantage of the present invention is that if the size of thememory block is selected correctly, then as processing continues acrossthe tile, the tiles tend to hit different cache lines. The selection ofthe size is also based on the compiler or the target machine, i.e. themachine on which the software is to be run. In one embodiment, the sizeof the buffer is selected with the zero band to be not a multiple of acache association size. If it is at an off multiple, there is betterchance to stay within the cache. With proper selection of size, there isa greater likelihood that locally used portions of the tile may be keptwithin the cache without reusing the same cache lines again, which isdesirable. Thus, the proper selection of the size of the buffer allowsfor better use of the cache.

The present invention also effectively handles edge cases. For handlingedges, there is no need to check if processing of an edge is occurringbecause the values outside the tile in the buffer at the edge aredefined to be a predetermined value. In other words, based on thepointer value, those coefficient values needed for a context that falloutside the tile are chosen to be the predetermined value. In oneembodiment, the predetermined value is zero. If mirroring is used, thesevalues may not be the same for all tiles. Thus, the tile is padded withzero coefficients filling the remaining space in the buffer to enablethe off edge conditions to be correct.

Note that in an alternative embodiment some of the values outside thetile may have values set to non-zero.

Common Occurrence Context Modeling

In the present invention, a context model uses a coefficient'sneighborhood and generates a context and a bit, which are sent to anentropy coder. A frequently used context (greater than 15-25% of thetime) is likely to have “runs” where the same context occurs frequently.

The FSM decoder, receiving the same context as an input, performs afeedback loop in response to each context. The feedback loop includesoperations such as shifting bits, updating a register, etc. The feedbackloop and the operations associated with it are time consuming. It isdesirable to perform the feedback loop only once for all of therepetitive contexts in a row, if possible, to reduce this overhead.

In one embodiment, the same context may occur typically in threesituations:

1) In performing Look Ahead as described in FIG. 16, Table 7 D2, andTable 8 C2, D2 can be determined for the current position to the edge ofthe image. This may be many look ahead intervals. If the previous C2 bitwas zero, C2 will use the same context for all the look ahead intervalsas long as the decoded bits are zero.

2) After the look Ahead for the TT-transform failed, referring to FIG.17 and Table 8, C3 for up to 16 bits, if the neighborhood is zero, thesame context is used as long as the decoded bits are zero; and

3) With respect to FIG. 19 and Table 8 C5, if neighborhood is zero, thesame context is sued as long as the decoded bits are zero.

Storing previously decoded bits using runcounts of coefficients that arezero allows rapid determination of nonzero neighborhood bits, allowingruns that can be the same context to be determined.

In the present invention, the decoder operates speculatively in that isit indeterminate whether one of the contexts is occurring more oftenuntil a previous consecutive number of these contexts has occurred. Thepresent invention examines the bitstream and determines whether therecould have been a predetermined number of the same context in a row, andif so, the multi-step process of updating the FSM coder and the contextmodel separately for all of the predetermined number of contexts isavoided and replaced with a single update. Thus, the FSM coder can skipahead the predetermined number of positions in the bitstream. Similarly,the context model skips ahead.

FIG. 34 illustrates one embodiment of the present invention. Referringto FIG. 34, FSM coder 3400 includes a lookup table (LUT) 3401 coupled toreceive the current FSM state 3410, the current probability state 3420for the context, and bits 3430 in the encoded bit stream that are beingdecoded. If the same context occurs multiple times in a row, then thebit is decoded for each occurrence of the context. If the actualbitstream matches a pattern of bits corresponding to the current FSMstate and an initial probability that indicate that a predeterminednumber (e.g., 5) of the same probability-class decisions will occur in arow, then the decoder performs the predetermined number of operations ina single operation, including updating the FSM state and the contextmodel, including their pointers.

Referring back to FIG. 34, the LUT 3401 is for n bits and is coupled toreceive current FSM state 3410 and an initial probability 3420. Based onthese inputs, a bitstream maximum 3412 and a bit stream minimum 3411 areoutput. These two bit stream outputs are compared to actual bitstream3430 using a comparator 3402. If actual bitstream 3430 is less than orequal to bitstream maximum 3412 and greater than or equal to bitstreamminimum 3411, then the output of comparator 3402 indicates that thespeculative decode may occur (e.g., its output is set to yes (Y));otherwise, the output of comparator 3402 indicates that the speculativedecode should not occur.

The shift output of LUT 3401 is coupled to a bit shifter that shifts inthe actual bit stream 3430. The bit shifter also receives the yes/nooutput of comparator 3402 as an enable signal. The next FSM state issent to an FSM state register which feeds back to provide the currentFSM state 3410. The FSM state register also receives the yes/no outputof comparator 3402 as an enable signal. The next probability output fromLUT 3401 is sent to the context memory storing the context along withthe yes/no output of comparator 3402 which acts as a right enable. Notethat the yes/no output of comparator 3402 is also sent to context model.

Note that when the speculative decode does not occur, the bits in actualbitstream 3430 are decoded in the normal, bit-by-bit fashion describedabove.

The bitstream maximum 3412 and minimum 3411 are used, instead of onebitstream, to compensate for the fractional bit case where the encoderdoes not output a bit for each FSM state. In this case, other bits mayhave to be examined in actual bitstream 3430. That is, in response to ann-bit run, the encoder generates a bit stream and changes to the FSMstate depending on the bits after the n-bit run. The encoder eventuallyoutputs bits in the encoded bitstream after the run of identicalcontexts which were based in part on that original run of contexts. Thebitstream maximum and minimum ensures that these bits are taken intoaccount.

In one embodiment, the lookup table also outputs the next FSM state, thenext probability estimate and the shift indication that specifies theamount of bits to shift in the actual bitstream that is incoming. Notethat any or all of these may be provided by separate tables.

In an alternative embodiment, a single lookup table, such as LUT 3501 ofFIG. 35, may receive the current FSM state, the initial probability andthe bitstream. In response to these inputs, the table might output a y/nindication, or an indication of the number of times the contextoccurred. Note that in this embodiment, instead of a fixed run length(e.g. 4), the run length is variable 0,1,2,3,4,5,6,7. However, oneproblem with this embodiment is that the table is bigger.

Distortion Spreading Over Tiles

In one embodiment of the present invention, tags are included in theencoded data stream. For example, one tag indicates the number of bitsthat are encoded at each importance level, summed over all the tiles.This is the BVI tag. This tag can be used to achieve fixed-sizedquantization for the decoded data having equal fidelity or quality ineach tile.

The BVI tag relates the number of bits to importance levels on animage-wide basis. This optional tag is used in the main header. The sizeof this variable-length tag depends on the number of importance levelsenumerated by the encoder. BVI: Table 11 shows the size and values forthe tile length main header parameters. TABLE 11 Bits versus importancelevels values Size Parameter (bits) Values BVI 16 0xff61 Lbvi 1610-65535 Cbvii 8  1-255 Ibvii 16  0-65535 Pbvii 32  0-(232 − 1) res 8  0(if necessary)Lbvi: Length of tag in bytes, not including the marker (the length iseven). Cbvi^(i): This signals which component data is being described.This Cbvi parameter, along with Ibvi and Pbvi, form a record that isrepeated for every component and importance level chosen to bedescribed. The tags are in order, with all importance-level descriptionsin the first component followed by those for the next component and soon. Ibvi^(i): The number of the importance level, in the currentcomponent, encoded by the number of bytes in Pbvii. This number (ornumbers) is selected at encode time to communicate interesting points inthe rate-distortion curve. This Ibvi parameter, along with Cbvi andPbvi, form a record that is repeated for every component and importancelevel described. Pbvi^(i): Number of bytes in the coded file thatinclude the main and tile headers and all data that relate to the numberof importance levels in Ibvii. This Pbvi parameter, along with Cbvi andIbvi, form a record that is repeated for every component and importancelevel described. res: A filler byte of zeros that is placed at the end,as needed.

To decode data to a fixed size representation with a fixed fidelitygiven that the data was encoded with tiles, the present invention usesimportance level entry points. At a minimum, each tile has one entrypoint. These are specified in the tags described herein. Note that wherea constant distortion over all tiles is not necessary, a fixed-ratequantization can be obtained by decoding a particular number of bytes ata particular resolution for each tile. However, this does not ensureeach tile has the same fidelity, just that each tile has the same amountof data.

In one embodiment of the present invention, the decoding maintains aconstant distortion by quantizing at the same importance levels in eachtile. Note that this may result in different amounts of data for eachtile; however, the fidelity of each tile is the same. Thus, the presentinvention provides for decoding so that the same quality of distortionis in each tile with the decoded bits being distributednon-proportionally (non equally) among the tiles. In this way, aconstant distortion over the entire image may be obtained.

As an example of why tiles might have different amounts of data for thesame fidelity, one tile might contain a complex natural image requiringa lot of coded data, while another tile might be a blank regionrequiring little data.

As discussed above, the BVI tag may be used to obtain the number ofbytes for a given importance level across the entire image. In otherwords, the BVI tag indicates how many bytes in an importance level.

In order to obtain the same quality of distortion across each tile, thepresent invention determines the number of bytes used to encode eachimportance level over the n tiles. This information may be determinedfrom the BVI tag. If a predetermined number, x, bytes are desired in thedecoded image, the bytes of each importance levels are added up and adetermination is made of where to stop in the importance level for eachtile. In other words, only a certain number of bytes are decoded toobtain the same fidelity, and decoding is stopped at the same importancelevel for every tile.

For example, if the BVI indicated the following summation of bits acrossthe entire image for the importance levels (the 16,011 bits listed nextto importance level 12 indicate the total number of bits for importancelevels 12 and 13) shown: Importance Level No. of Bits 13 4,096 12 16,01111 40,000 10 100,000 9 250,000 8 500,000 7 1,000,000 6 2,500,000 55,500,000

For example, only 750,000 bits may be allowed in the decoded image, thenall that can be decoded (as the 1,000,000 bits tested with importancelevel 7 includes the 500,000 bits of importance levels 8-13) is throughimportance level 8 and half of importance level 7.

The present invention provides for numerous alternatives for decidingwhere to stop decoding in the data. One could decode importance levelsuntil the data ran out. For example, in the example above, one coulddecide to decode from importance level 13 down to half way throughimportance level 7. In another embodiment, a percentage could bedetermined for each level and then that percentage may be applied toeach of the importance levels. For instance, if only half of level 7 wasto be decoded as determined above, this fifty percent could be appliedto all of the levels until the data ran out, instead of only one half ofthe data after importance level 8 would be decoded. Thus, in this case,the total percentage between the budgeted amount of decoded data and thedata to be decoded dictates the decision on where to stop decoding oneach importance level. Thus, the present invention provides forproviding a fixed size representation with the same fidelity across eachtile. In other words, the target size of the image is fixed with thedifferent segments contained therein having different data rates.

The application of the present invention is affected by the alignment.In one embodiment, the data is in a normalized alignment, such as shownin FIG. 39. In an alternate embodiment, the data is in the pyramidalalignment, such as shown in FIG. 40. Although the above example isperformed with a normal alignment, the present invention can beperformed while decoding data encoded with a pyramidal alignment toobtain a fixed size image with the same fidelity across all tiles, witha normalized alignment. The use of the normalized alignment produces thebest quality with respect to squared error. Once again, the data in theBVI tag indicates the total number of bytes from which the presentinvention subtracts the number of bytes that are allocated to thedecoded image. If the data is in a pyramidal alignment with entrypoints, the entry points allow skipping ahead and truncating the correctamount of data at each segment. In one embodiment, truncation isperformed in a normalized fashion, resulting in the best MSE.

For instance, based on the determination of the amount of data that isallowed through the use of the BVI tag, a decision to truncate the datamay be made across importance levels of each pyramid level where thedata is in a pyramidal alignment. For instance, referring to FIG. 40,the data may be decoded from importance level 30 to importance level 25and then truncated from importance level 24 to 20. Decoding of the datawould also occur from importance level 19 to importance level 15followed by truncation from importance level 14 to 10, and decoding ofthe data would occur from importance level 9 to importance level 5 withtruncation from importance level 4 to 0. Thus, for each of the frequencybands at different levels, the same data is being truncated, and in thisformat, it is known what impact such truncation will have. This isequivalent to truncating the data in the normalized alignment at level5.

In another scenario, if the alignment is normal, pyramidal quantizationis possible by similar means. If there is an entry point at thebeginning of each importance level, then each importance level can bedecoded up to the pyramidal levels of interest. For example, in FIG. 40,if the image resolution is to be reduced by two in each dimension(pyramidal level 1 is to be quantized), each the data corresponding topyramidal importance level 3 and 2 are decoded. When complete thedecoder drops to the beginning of the next importance level.

It should be noted that every frequency band may be set forth in apyramidal arrangement in which each frequency band follows the other.This does increase the number of importance levels dramatically.However, because each frequency band has a single width, it facilitatesthe truncation of data so that its affect on the rest of the image isbetter understood. Note that this would require a number of resets tobring one back to the beginning of each frequency band so as to allowone to truncate at the proper location.

Thus, in general, regardless of the alignment when its encoded, eithernormalized, pyramidal alignment, or any other alignment, the data may bedecoded to obtain a constant distortion across all tiles based oninformation in the tags. The data in the tags helps determine what datato decode so as to achieve a fixed target image with the same resolutionover all the tiles.

There can be a BVI for each color component and the user can decide howto allocate bits for each color component. The user can then use the BVIto determine the importance level to stop at for each component.Therefore, the amount of information in the components can be comparedand contrasted and decisions can be made on how to allocate bits amongthe components.

Therefore, the BVI tag allows the specification of multiple componentsto enable the selection of a percentage of bits from each of themultiple components for truncation. This ensures the distribution acrossall tiles regardless of the rate, with no equal rate or size per tile.

The data in a BVI tag can be synthesised, or supplemented, by the IET orIEM, and the ILT or ILM tag information. Each of these tags point toimportance levels in a tile. If they exist for all the tiles, this datacan be added to create BVI-like information.

Whereas many alterations and modifications of the present invention willno doubt become apparent to a person of ordinary skill in the art afterhaving read the foregoing description, it is to be understood that anyparticular embodiment shown and described by way of illustration is inno way intended to be considered limiting. Therefore, references todetails of various embodiments are not intended to limit the scope ofthe claims which in themselves recite only those features regarded asessential to the invention.

1. (canceled)
 2. (canceled)
 3. (canceled)
 4. (canceled)
 5. (canceled) 6.(canceled)
 7. (canceled)
 8. (canceled)
 9. (canceled)
 10. (canceled) 11.(canceled)
 12. (canceled)
 13. (canceled)
 14. (canceled)
 15. (canceled)16. (canceled)
 17. (canceled)
 18. (canceled)
 19. (canceled) 20.(canceled)
 21. (canceled)
 22. (canceled)
 23. (canceled)
 24. (canceled)25. (canceled)
 26. (canceled)
 27. (canceled)
 28. (canceled) 29.(canceled)
 30. (canceled)
 31. (canceled)
 32. (canceled)
 33. (canceled)34. (canceled)
 35. (canceled)
 36. (canceled)
 37. (canceled) 38.(canceled)
 39. (canceled)
 40. (canceled)
 41. (canceled)
 42. (canceled)43. (canceled)
 44. (canceled)
 45. (canceled)
 46. (canceled) 47.(canceled)
 48. (canceled)
 49. (canceled)
 50. (canceled)
 51. (canceled)52. (canceled)
 53. (canceled)
 54. (canceled)
 55. (canceled) 56.(canceled)
 57. (canceled)
 58. (canceled)
 59. (canceled)
 60. (canceled)61. (canceled)
 62. (canceled)
 63. (canceled)
 64. (canceled) 65.(canceled)
 66. (canceled)
 67. (canceled)
 68. (canceled)
 69. (canceled)70. A method for performing a reconstruction comprising the steps of:generating a reconstruction value; and clipping the reconstruction valueif the reconstruction value exceeds the bounds of neighboringcoefficients, where the reconstruction value is clipped to a coefficientvalue within the bounds of the neighboring coefficients of a neighboringcoefficient with a value closest to the reconstruction value.
 71. Themethod defined in claim 70 using a center sample as the reconstructionvalue if the subsequent reconstruction value does not exceed the boundsof neighboring coefficients.
 72. The method defined in claim 70 using asample value of one of a plurality of immediate neighbors of a centersample as the reconstruction value if the subsequent reconstructionvalue does not exceed the bounds of neighboring coefficients.
 73. Themethod defined in claim 72 wherein the sample value of the one of theplurality of the immediate neighbors has a value closer to the centersample than the other of the he plurality of the immediate neighbors.74. The method defined in claim 70 wherein the neighboring coefficientscomprise neighboring S coefficients.
 75. The method defined in claim 70where the reconstruction value is clipped to a coefficient value of aneighboring coefficient with a value closest to the reconstructionvalue. 76-86. (canceled)